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The  Military  Operations  Research  Society 

The  purpose  of  the  Military  Operations  Research  Society  is  to  enhance  the  quality  and 
effectiveness  of  classified  and  unclassified  military  operations  research.  To  accomplish  this 
purpose,  the  Society  provides  media  for  professional  exchange  and  peer  criticism  among 
students,  theoreticians,  practitioners,  and  users  of  military  operations  research.  These  media 
consist  primarily  of  the  traditional  annual  MORS  symposia  (classified),  their  published 
proceedings,  special  mini-symposia,  workshops,  colloquiaand  special  purpose  monographs.  The 
forum  provided  by  these  media  is  directed  to  display  the  state  of  the  art,  to  encourage  consistent 
professional  quality,  to  stimulate  communication  and  interaction  between  practitioners  and  users, 
and  to  foster  the  interest  and  development  of  students  of  operations  research.  In  performing  its 
function,  the  Military  Operations  Research  Society  does  hot  make  or  advocate  official  policy  nor 
does  it  attempt  to  irifluence  the  formulation  of  policy.  Matters  discussed  or  statements  made 
during  the  course  of  its  symposia  or  printed  in  its  publications  represent  the  positions  of  the 
individual  participants  and  authors  and  not  of  the  Society. 

The  Military  Operations  Research  Society  is  operated  by  a  Board  of  Directors  consisting  of 
30  members,  28  of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four  years.  The 
persons  nominated  for  this  election  are  normally  individuals  who  have  attained  recognition  and 
prominence  in  the  field  of  military  operations  research  and  who  have  demonstrated  an  active 
interest  in  its  programs  and  activities.  The  remaining  two  members  of  the  Board  of  Directors 
are  the  Past  President  who  serves  by  right  and  the  Executive  Director  who  serves  as  a 
consequence  of  his  position.  A  limited  number  of  Advisory  Directors  are  appointed  from  time 
to  time,  usually  a  1-year  term,  to  perform  some  particular  function.  Since  a  major  portion  of 
the  Society’s  affairs  is  connected  with  classified  services  to  military  sponsors,  the  Society  does 
not  have  a  general  membership  in  the  sense  that  other  professional  societies  have  them.  The 
members  of  MORS  are  the  Directors,  persons  who  have  attended  a  MORS  meeting  within  the 
past  three  years  and  Fellows  of  the  Society  (FS)  who,  in  recognition  of  their  unique  contributions 
to  the  Society,  are  elected  by  the  Board  of  Directors  for  life. 

MORS  is  sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director  Assessments  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Modeling,  Simulation  and  Analysis,  Deputy  Chief  of  Staff,  Plans  and 
Operations,  Headquarters,  US  Air  Force 

•  The  Commanding  General,  Marine  Corps  Combat  Developments  Command 

•  The  Director  of  Force  Stracture,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 
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Chapter  I 
Introduction 


1.0  Purpose 

The  requirement  to  measure  the  effects 
of  C^IEW  systems  on  battle  outcome  contin¬ 
ues  to  challenge  the  operations  research 
community.  This  community  has  long  recog¬ 
nized  the  difficulty  in  measuring  C^IEW 
system  effectiveness  in  terms  of  battlefield 
attrition  and  end  strengths.  However,  with 
the  recent  political  changes  in  the  world,  the 
changing  role  of  US  military  to  a  power 
projection  force,  and  the  lessons  of  Desert 
Storm,  the  need  to  analyze  and  understand 
the  effects  of  C^IEW  systems  has  never  been 
greater.  The  doctrinal  evolution  of  military 
operations  has  heightened  the  need  for  cam¬ 
paign  level  C^IEW  measures  of  effectiveness 
(MOE). 

The  purpose  of  this  report  is  to  docu¬ 
ment  the  results  of  a  MORS  sponsored 
workshop  held  at  Fort  Leavenworth,  Kan¬ 
sas,  in  October  1992  to  address  the  develop¬ 
ment  and  application  of  C^IEW  MOE 
throughout  DoD. 

1.1  Background 

Research  efforts  directed  at  the  develop¬ 
ment  and  application  of  C^IEW  measures  of 
effectiveness  have  experienced  periodic 
surges  in  interest  and  activity  followed  by 
relatively  dormant  periods.  For  example, 
within  the  U.S.  Army,  the  period  from  the 
mid-  to  late- 1970s  saw  development  of  new 
automated  command  and  control  systems, 
such  as  the  Tactical  Fire  Direction  System 
(TACFIRE)  and  the  Tactical  Operations 
System  (TOS).  These  systems  were  being 
developed  with  all  haste  and  they  held  great 


promise  for  increasing  force  effectiveness. 
Along  with  these  developments,  of  course, 
came  the  requirements  for  operational  test 
and  evaluation  and  for  the  conduct  of  Cost 
and  Operational  Effectiveness  Analysis 
(COEA).  This  led  to  a  flurry  of  activity  to 
develop  C^IEW  measures  of  effectiveness 
Unfortunately,  the  tests  and  the  COEAs 
failed  to  show  an  increase  in  effectiveness 
which  could  be  attributed  to  these  systems. 
This  was  due,  in  part,  to  the  inability  to 
measure  the  effectiveness  of  these  systems 
either  in  the  tests  or  in  the  combat  models 
used  in  the  COEAs. 

This  inability  to  measure  effectiveness 
contributed  significantly  not  only  to  the 
cancellation  of  the  TOS  program  by  Con¬ 
gress,  but  also  to  a  marked  decline  in  the 
Army’s  efforts  to  develop  C^IEW  measures 
of  effectiveness. 

By  1985  the  rapid  growth  of  computer 
technology  had  greatly  expanded  the  capabil¬ 
ities  of  automated  C^lEW  systems.  Com¬ 
mand  and  control  systems  became  larger, 
more  automated,  and  more  integrated  with 
communications,  intelligence,  and  electronic 
warfare  systems.  In  light  of  these  develop¬ 
ments,  there  was  a  renewed  interest  in  the 
development  of  a  practical  set  of  measures 
of  effectiveness  and  evaluation  methodolo¬ 
gies  to  support  military  decision  makers 
throughout  the  DoD  in  their  assessments  of 
command  and  control  and  C^IEW  systems. 
Recognizing  this  need,  MORS  sponsored  a 
C^  Evaluation  Workshop  in  January  1985  to 
develop  a  conceptual  model  for  C^  and 


1 


provide  an  analysis  framework  for  measur¬ 
ing  the  performance  and  effectiveness  of 
systems  within  that  model.  This  framework 
was  called  the  Modular  Command  and  Con¬ 
trol  Evaluation  Structure  (MCES).  The 
product  of  the  workshop  was  a  draft  report 
which  was  subsequently  finalized  and  pub¬ 
lished  by  MORS  in  June  1986.  Also  in 
1986,  the  application  of  MCES  to  a  series  of 
military  C^IEW  systems  was  demonstrated 
and  results  published  by  the  Naval  Postgrad¬ 
uate  School  in  September  1986.  From  this 
point  through  1991  few  new  developments 
of  C^IEW  measure  of  effectiveness  are 
documented. 

In  November  of  1911,  the  US  Army 
TRADOC  Analysis  Command  (TRAC),  at 
the  direction  and  under  the  sponsorship  of 
Mr.  Walt  Hollis,  Deputy  Under  Secretary  of 
the  Army  for  Operations  Research 
[DUSA(OR)],  conducted  a  C^I  Modeling 
and  Simulation  Conference.  The  purpose  of 
this  conference  was  to  share  what  C^I  activi¬ 
ty  was  in  progress  in  the  Army  training, 
testing,  analysis,  and  research  communities 
and  to  determine  what  needed  to  be  done  to 
collectively  improve  these  activities.  One 
major  recommendation  from  this  conference 
was  that  more  work  was  needed  in  C^I  mea¬ 
sures  of  effectiveness  development  and  that 
this  work  needed  to  result  in  more  consistent 
measures  across  these  communities.  As  a 
result  of  this  conference  recommendation, 
Mr  Hollis,  as  a  MORS  sponsor,  asked 
MORS  officials  to  consider  sponsoring  a 
workshop  to  address  C^IEW  MOE  develop¬ 
ment.  TRAC  volunteered  to  host  this  work¬ 
shop  for  MORS  at  Fort  Leavenworth.  The 
workshop  was  conducted  from  20  through 
23  October  1992  and  was  attended  by  some 
one  hundred  trainers,  testers,  analysts,  and 
researchers  from  throughout  DoD  and  the 
civilian  community. 


1.2  Workshop  Organization 

The  attendees  were  divided  into  four 
working  groups  which  reflected  the  function¬ 
al  areas  of  the  military  OR  community: 
Operations,  Plans,  Training;  Analysis;  Test 
and  Evaluation;  and  Research.  Additionally, 
a  synthesis  group  participated  in  the  discus¬ 
sions  with  all  working  groups.  The  synthe¬ 
sis  group  had  a  member  in  each  working 
group  to  identify  common  threads  among  the 
groups  and  to  summarize  the  main  points  of 
the  discussions. 

The  workshop  began  the  evening  of  20 
Oct  92  with  a  dinner  and  an  address  by 
COL  Jim  McDonough,  Director  of  the 
School  of  Advanced  Military  Studies, 
USACGSC.  COL  McDonough  discussed  the 
concept  of  a  campaign  and  provided  some 
insights  on  the  changes  being  made  to  the 
Army’s  doctrine.  The  intention  of  his  dis¬ 
cussion  was  to  focus  the  group  on  relating 
MOP,  MOE  and  MOFE  to  the  theater  cam¬ 
paign  level.  Appendix  A  provides  excerpts 
from  this  presentation. 

The  workshop  ran  as  a  series  of  four 
working  sessions,  with  each  session  discuss¬ 
ing  one  of  the  four  objectives.  Each  session 
began  with  an  address  by  an  invited  speaker. 
After  each  address,  the  participants  divided 
into  their  working  groups  to  address  the 
appropriate  objective.  At  the  end  of  the 
working  session,  the  participants  assembled 
to  hear  each  Working  Group  Chair  report  on 
the  findings  of  their  working  group. 

The  keynote  address,  which  also  acted  as 
the  address  for  the  first  session,  was  pre¬ 
sented  by  LTG  Wilson  A.  Shofther,  Com¬ 
mander,  USA  Combined  Arms  Command. 
General  Shoffner  shared  his  views  on  com¬ 
mand  and  control,  and  the  effect  of  recent 
world  changes  on  the  conduct  of  campaigns. 
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General  Shoffner  provided  a  superb  discus-  how  to  implement  MCES  on  a  real  sys- 

sion  of  the  dynamics  of  the  future  battle-  tem.Bccerpts  from  his  presentation  are 

field,  the  role  of  battle  command  in  a  cam-  located  in  appendix  C.  Dr.  Dubin  addressed 

paign,  and  the  relationship  between  com-  the  definitional  context  of  C^I  measures, 

mand  and  control.  He  emphasized  that  typical  measures,  teaching  command  and 

command  and  control  are  not  one  word.  control,  and  the  challenges  associated  with 

Command  is  an  art  that  is  the  commander’s  this  field.  Highlights  of  his  presentation  are 

business,  while  control  is  a  science  that  is  in  appendix  D.  Mr.  Brady’s  address  provid- 

the  staffs  business.  A  key  issue  that  he  ed  an  historical  perspective  with  the  key 

raised  was  to  give  up  on  modeling  the  intu-  trends  and  key  issues,  and  presented  some 

itive  aspect  of  the  decision  making  process  thoughts  on  emerging  tools  and  technolo- 

and  use  a  man-in-the-loop  to  represent  the  gies.  Receipts  are  in  appendix  E. 

decision  maker.  Appendix  B  contains  ex¬ 
cerpts  from  LTG  Shoffner’ s  presentation.  1.3  Workshop  Objectives 

There  were  four  objectives  for  the  work- 
The  other  plenary  presentations  were  shop.  Each  working  group  was  asked  to 

made  by  Dr.  Mike  Sovereign,  Dr.  Hank  address  the  questions  associated  with  these 

Dubin,  and  Mr.  Ed  Brady.  Dr.  Sovereign  objectives.  The  objectives  and  questions  are 

reviewed  MCES,  which  was  developed  listed  in  the  Table  I-l  below: 

during  the  1985  MORS  Workshop.  He  used 
the  VTOL  UAV  as  n  example  to  show 

Table  1-1.  Workshop  Objectives 

Objective  1:  Definition  of  Force-level  C^IEW 

•  What  is  the  nature  of  force-level  C^IEW  within  a  Campaign? 

•  What  are  terminology  differences  among  services? 

•  What  are  key  essential  elements  of  analysis  (EEA)  associated  with  force-level  C^IEW? 

Objective  2:  Description  of  Force-level  C^IEW  Measures 

•  What  are  C^IEW  MOP,  MOE  and  MOFE  and  the  relationship  among  them? 

•  What  are  the  differences  in  perspective  between  testers,  trainers,  analysts,  research¬ 
ers  and  developers? 

•  What  approaches/methodologies  can  be  used  to  develop  these  measures? 

Objective  3:  Quantifying  C^IEW  Measures  and  Answering  C^IEW  EEA 

•  What  Capability  currently  exists  to  collect  and  quantify  C^IEW  measures? 

•  What  are  the  problems  associated  with  this  capability  and  how  can  they  be  over¬ 
come? 

•  How  can  testers,  trainers,  analysts,  researchers  and  developers  best  work  together? 

Objective  4:  Tools  and  Methodologies  to  support  C^IEW  Evaluation 

•  What  deficiencies  exist  in  the  tools  and  methodologies  used  to  measure  C^IEW? 

•  How  can  emerging  technology  overcome  these  deficiencies? 

•  How  can  we  work  together  to  improve  our  tools  and  methodologies? 
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Chapter  n 

Definition  of  Force-level  C^IEW 


2.0  Introduction 

To  proceed  through  the  process  of  de¬ 
scribing  force-level  C^IEW  measures  of 
effectiveness,  it  is  important  to  first  define 
what  is  meant  by  a  campaign  and  force-level 
C^IEW.  The  working  groups  identified  nu¬ 
merous  descriptions  from  a  multitude  of 
perspectives.  One  description  for  a  campaign 
from  the  draft  Joint  Pub  5-00.1  says  that 
"campaigns  represent  the  art  of  linking  bat¬ 
tles  and  engagements  in  an  operational 
design  to  accomplish  strategic  objectives. . . " 
In  this  context  a  campaign  can  then  be 
defined  as  "a  series  of  related  military  oper¬ 
ations  aimed  to  strategic  objectives  within  a 
given  time  and  space. " 

With  the  role  of  the  military  focusing 
more  on  operations  other  than  war,  i.e., 
peacekeeping  activities,  relief  activities  such 
as  Operation  Provide  Comfort,  and  the  relief 
effort  in  Somalia,  the  characteristics  of  a 
campaign  may  change  and  likewise  change 
the  nature  of  C^IEW  within  a  campaign.  As 
noted  in  COL  McDonough’s  dinner  speech 
(see  appendix  A),  the  Army  is  rewriting  its 
doctrine  to  incorporate  the  entire  spectrum 
of  military  operations,  from  peacekeeping 
activities  to  total  war.  Any  effort  in  describ¬ 
ing  C^IEW  measures  of  effectiveness  must 
ultimately  account  for  the  variety  of  mis¬ 
sions  that  can  be  undertaken  by  the  military 
in  a  post-Cold  War  environment.  However, 
for  purposes  of  narrowing  the  scope,  this 
workshop  focused  on  combat  operations 
with  forces  already  in-theater. 


2.1  The  Role  of  C^IEW  in  a  Campaign 

A  campaign  provides  a  framework  for 
discussing  force-level  C^IEW.  Today,  any 
campaign  conducted  by  U.S.  military  forces 
must  be  considered  a  joint  campaign  run  by 
a  CINC  or  Joint  Task  Force  Commander. 
Single  service  campaigns  are  not  a  reality 
any  more.  In  terms  of  unit  size,  force-level 
implies  corps  or  echelons  above  corps 
(EAC)  for  Army,  numbered  Air  Force,  and 
Battle  Group  or  Battle  Force  for  the  Navy. 

Joint  Pub  1-02  defines  command  and 
control  as  ".  .  .  the  exercise  of  authority  and 
direction  by  a  properly  designated  com¬ 
mander  over  assigned  forces  in  the  accom¬ 
plishment  of  a  mission.  Command  and 
control  functions  are  performed  through  an 
arrangement  of  personnel,  equipment,  com¬ 
munications,  facilities,  and  procedures 
employed  by  a  commander  in  planning, 
directing,  coordinating,  and  controlling 
forces  and  operations  in  the  accomplishment 
of  the  mission. "  This  definition  is  invariably 
included  at  the  beginning  of  any  document 
discussing  command  and  control.  This 
report  will  be  no  different  in  that  sense. 
What  is  important  from  this  definition  is  the 
understanding  of  the  role  command  and 
control  systems  in  supporting  the  command¬ 
er  to  accomplish  these  functions. 

From  Control  of  the  Joint  Force:  A  New 
Perspective,  "General  Depuy  reminds  us  that 
the  purpose  of  command  and  control  is  to 
produce  unity  of  effort  from  a  diversity  of 
means  and  that  the  key  ingredient  is  not  the 
automated  systems,  but  the  mind  of  the 
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commander."  The  focus  of  command  and 
control,  and  all  of  C^IEW,  must  be  to  better 
support  the  friendly  commander’s  decision 
making  process  and  to  degrade  that  of  the 
opposing  commander.  To  make  the  best 
decision,  the  commander  must  have  the  best 
information  available.  It  is  the  role  of  the 
C^IEW  to  provide  that  relevant  information 
in  a  timely,  accurate  manner. 

In  his  keynote  address  (Appendix  B), 
LTG  Sholfner  described  the  nature  of  the 
future  battlefield  as  having  the  potential  for 
an  accelerated  tempo.  Lethality  is  increas¬ 
ing;  mobility  is  increasing;  weapons  preci¬ 
sion  is  increasing;  and,  range  of  fires  is 
increasing.  The  acceleration  in  these  battle¬ 
field  dynamics  brings  an  increase  in  the 
amount  of  information  and  intelligence  that 
the  commander  must  comprehend  in  a  short 
period  of  time.  The  employment  of  multiple 
lethal  weapons  in  the  attack  will  require 
intricate  coordination  and  timing  that  only  a 
sophisticated  command  and  control  system 
can  provide.  Likewise,  dismption  of  the 
enemy’s  capability  could  allow  the  com¬ 
mander  to  control  or  slow  the  op  tempo 
when  beneficial  to  friendly  forces. 

While  command  is  primarily  an  art,  con¬ 
trol  is  primarily  a  science.  Intelligence  pro¬ 
vides  the  percqjtion  of  the  enemy,  while 
electronic  warfare  provides  the  means  to 
deny  the  enemy  an  accurate  perception  of 
friendly  forces.  Communications  provides 
the  glue  to  make  it  all  tie  together.  C^IEW 
as  a  whole  provides  the  grease  to  reduce  the 
friction  of  war,  across  concept  formulation, 
planning,  and  execution. 

2.2  Differences  in  Terminology 

One  of  the  hurdles  that  must  be  over¬ 
come  in  the  effort  of  describing  C^IEW  and 
C^IEW  measures  is  the  difference  in  termi¬ 


nology  among  the  services  and  the  tremen¬ 
dous  number  of  acronyms.  There  are  real 
differences  in  systems  among  the  services 
as  well  as  different  Intelligence  and  EW  sys¬ 
tems,  so  terminology  can  not  be  expected  to 
be  the  same.  Message  text  formats  and  data 
dictionaries  are  becoming  standardized  but 
procedures  are  still  unique,  often  to  specific 
commands  and  commanders.  This  affects  the 
problem  of  interoperability  in  both  the  real 
and  the  modeling  worlds. 

No  adequate  joint  theater-level  campaign 
C^EEW  model  exists  although  PACOM  may 
be  beginning  to  build  one.  Joint  exercise 
support  systems  are  being  cobbled  together: 
Aggregate  Level  Simulation  Protocol 
(ALSP)  and  Data  Bus,  for  example,  are 
linkages  of  major  war  games  that  could  be 
used  to  examine  C^IEW. 

2.3  Essential  Elements  of  Analysis 

Not  only  are  there  differences  among  the 
services  in  terminology  of  C^IEW  systems, 
there  are  also  differences  among  the  services 
in  analysis  terminology.  The  term,  Essential 
Elements  of  Analysis  (EEA),  is  not  often 
used  outside  of  the  Army.  One  suggested 
term  is  ‘critical  analysis  questions’  as  an 
equivalent,  but  there  is  difficulty  even  m 
defining  those.  One  question  which  recurs  is 
whether  EEA  are  exclusively  developed  to 
support  analysis  for  acquisition  or  for  opera¬ 
tions,  training,  etc.  Army  EEA  seem  to  be 
designed  for  use  across  acquisition  of  all 
types  of  systems.  For  C^IEW  analysis,  the 
proposed  essential  elements  of  analysis  are 
summarized  in  Table  11- 1.  These  emphasize 
the  key  decisions  that  are  made  in  the  cam¬ 
paign  and  the  information  required  to  sup¬ 
port  the  decisions.  Unfortunately,  there  is  no 
reference  document  for  the  campaign  deci¬ 
sions! 
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Table  11-1.  Essential  Elements  of  Analysis  for  C^IEW 

•  What  is  the  applicability  of  information  at  each  echelon?  How  does  a  system  get  the  right  da¬ 
ta/information  to  the  right  person  at  the  right  time? 

•  What  are  system  classification  issues  that  need  to  be  overcome  to  complete  analysis? 

•  What  does  a  C^IEW  system  need  to  do  and  how  well  does  it  need  to  do  it? 

•  What  are  the  key  decisions  that  affect  the  campaign  outcome? 

•  Does  the  proposed  change/system/process  support  the  commander's  needs? 

•  What  are  the  key  information  elements  needed  by  the  joint  component  commanders  and  their  staffs? 

•  When  is  information  needed? 

•  Does  missing,  late  or  inaccurate  information  change  key  decisions? 


The  view  of  C^IEW  EEA  shown  in 
Table  2-1  suggests  that  the  C^IEW  process 
is  a  hierarchy  with  command  at  the  top, 
control  in  the  middle,  and  communications 
tying  sensors  to  information,  intelligence, 
and  EW  systems  at  the  bottom.  With  this 
structure,  some  EW  functions,  such  as 
jammers  and  radar  warning  receivers  on 
aircraft,  reside  outside  of  a  force-level 
discussion. 


An  important  question  that  remains 
unanswered  is:  at  what  level  of  resolution 
should  the  decisions  be  represented  in  an 
analysis  of  force-level  C^BEW?  No  concur¬ 
rence  was  reached  with  some  believing  that 
an  aggregate  level  was  adequate,  while 
others  believed  that  much  more  detailed 
modeling  would  be  needed,  for  example  that 
logistics  decisions  would  be  necessary. 
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Chapter  m 

Description  of  Force-level  C^BEW  Measures 


3.0  Introduction 

The  MORS  workshop  on  MOE  in 
1985  laid  the  foundation  for  much  of  the 
content  of  this  chapter.  One  of  the  results 
of  that  workshop  was  the  Modular  Evalu¬ 
ation  Structure  (MCES),  a  methodology  to 
identify  measures  that  are  appropriate  for 
analysis.  The  MCES  approach  was  de¬ 
scribed  in  a  plenary  presentation  by  Dr. 
Mike  Sovereign.  His  presentation  is  summa¬ 
rized  in  Appendix  C.  Key  elements  of  the 
work  from  the  1985  workshop  include  a  set 
of  definitions  for  measures  and  the  establish¬ 
ment  of  a  relationship  among  those  mea¬ 
sures.  This  relationship  provides  the  frame¬ 
work  in  which  an  analysis  of  a  system 
can  occur. 

3.1  Definitions 

The  1985  workshop  provided  the  initial 
set  of  definitions  for  measures.  Ingram 
and  Short  modified  these  definitions  slightly 
in  their  1991  paper,  Command  and  Control 
Measures:  A  Proposed  Approach.  These 
definitions  are  modified  and  extended  to 
apply  to  C^iEW  systems: 

CflEW  system.  A  C^IEW  system  has 
three  components:  physical  entities,  struc¬ 
tures,  and  processes. 

(1)  Physical  entities  refer  to  equip¬ 
ment,  software,  facilities,  and  people. 

(2)  Structures  identify  the  arrange¬ 
ment  and  mterrelationships  of  physical 
entities,  procedures,  protocols,  concepts  of 
operation,  and  mformation  patterns.  Such 


arrangements  are  often  spatial  and  temporal. 

(3)  Processes  are  what  the  system  is 
doing  and  reflect  the  functions  carried  out 
by  the  system. 

Dimensional  parameter.  A  property  or 
characteristic  inherent  in  the  physical  entities 
whose  value  determines  system  behavior  and 
the  structure  under  question  even  when  at 
rest.  Examples  are  size,  weight,  capacity, 
and  number  of  pixels.  These  measures  are 
obviously  scenario  independent. 

0IEW  measure  of  performance  (C^IEW 
MOP).  A  measure  of  an  attribute  of  system 
behavior.  Examples  are  throughput  and 
error  rate.  C^IEW  MOP  are  related  to  the 
inherent  parameters  (physical  and  structur¬ 
al),  are  internal  to  the  system  being  ana¬ 
lyzed,  and  are  scenario-independent. 

CflEW  measure  of  effectiveness  (C^IEW 
MOE).  A  measure  of  how  a  C^IEW  system 
affects  the  other  entities  within  an  operation¬ 
al  environment.  Examples  are  reaction  time, 
number  of  targets  detected,  and  susceptibili¬ 
ty  to  deception).  C^IEW  MOE  are  measured 
relative  to  some  standard  or  baseline,  which 
is  often  implicit  (e.g.,  how  a  perfect 
system  would  perform).  C^IEW  MOE  are 
scenario-dependent. 

Measure  offeree  effectiveness  (MOPE). 
A  measure  of  how  the  force  performs  its 
mission.  These  are  typically  measures  that 
are  used  for  weapon  system  analysis,  for 
example,  loss  exchange  ratios  (LERs),  or 
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number  of  weapons  destroyed.  As  with 
C^IEW  MOE,  MOPE  are 
scenario-dependent. 

Boundary  of  a  CflEW  system.  The  delin¬ 
eation  between  the  C^IEW  system  being 
studied  and  the  environment.  Within  this 
definition,  measures  can  shift  from  one  type 
to  another  depending  on  the  context  of  the 
C^IEW  system  boundary.  Figure  ni-l  graph¬ 
ically  shows  the  relationship  of  the  measures 
defined  above.  This  diagram  is  typically 
referred  to  as  the  ‘onion’  diagram,  because 
each  layer  can  be  pealed  back  to  reveal  the 
next  lower  level. 

3.2  Measure  Relationships 
NOTE:  For  purposes  of  generalization,  the 
term.  Measures  of  Merit  (MoMs),  will  be 
used  to  refer  to  all  measures. 


There  are  several  ways  to  view  C^IEW 
Measures  of  Merit  and  their  relationship. 
Each  viewpoint  provides  a  different  perspec¬ 
tive  toward  the  measures.  Three  viewpoints 
are  presented  here:  the  ’onion’  diagram,  the 
hierarchical  relationship,  and  the  properties 
measures. 

The  ‘onion’  diagram  of  the  MCES  can 
be  used  as  a  tool  for  bounding  the  problem. 
But,  the  boundaries  of  the  system  must  be 
drawn  carefully  for  the  definitions  to  work. 
That  must  be  accomplished  within  the  con¬ 
text  of  a  particular  analysis  to  answer  the 
EEA.  The  perspective  from  which  the  analy¬ 
sis  occurs  may  change  a  measure  of  effec¬ 
tiveness  to  a  measure  of  performance.  For 
example,  depending  on  the  level  of  the 
analysis  to  be  examined,  measures  which  are 
normally  MOEs  for  lower  level  examina- 


Environment 


E  =  Effectiveness 
F  =  Force  Effectiveness 


Figure  lli-1.  'Onion  Diagram' 
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tions  may  become  assumptions  at  higher 
levels. 

The  hierarchical  relationship  of  mea¬ 
sures  of  merit  suggests  an  ordered  structure 
that  allows  the  linking  of  lower  level  mea¬ 
sures  to  higher  level  measures.  In  this  rela¬ 
tionship,  measures  of  effectiveness  (MOE) 
can  be  linked  to  certain  measures  of  perfor¬ 
mance  (MOP)  which,  in  turn,  can  be  linked 
to  specific  dimensional  parameters  (DP).  To 
use  this  relationship  effectively,  the  lower 
measures  can  be  analyzed  to  provide  some 
indication  of  the  higher  level  measures.  If  a 
direct  relationship  can  be  established  be¬ 
tween  lower  and  higher  measures,  conclu¬ 
sions  can  be  reached  about  system  effective¬ 
ness  through  the  analysis  of  system  perfor¬ 
mance.  Care  must  be  taken  to  appropriately 
extend  a  cause  and  effect  relationship  to  this 
hierarchical  structure.  Figure  in-2  depicts 
the  hierarchical  relationship  for  both  weapon 
systems  and  C^IEW  systems. 


Figure  III-2.  Hierarchical  Relationship 
of  Measures 


Measures  can  be  described  in  terms  of 
the  level  of  resolution  of  a  particular  proper¬ 
ty  of  the  C^IEW  system.  This  approach  dis¬ 
sects  a  system  in  terms  of  its  functionality  or 
purpose  and  the  properties  associated  with 
that  functionality.  Through  this  description. 


measures  of  merit  can  be  developed  and 
associated  with  the  key  properties  of  the 
system.  Table  ni-l  provides  an  example  of 
the  levels  of  a  system  property.  Although 
the  term  ’MOE’  is  used  in  an  analysis,  it  is 
often  unclear  which  property  level  is  being 
considered. 


Table  lil-1.  Levels  of  a  System  Property 


PROPERTY 

EXAMPLE 

1.  Family  Name 

Vulnerability 

2.  Surname 

Susceptibility  to 

Jamming 

3.  System  Reference 

VAV  Data  Link 

4.  Functional  Ref. 

Sense 

5.  Units 

BITS/Sec 

6.  Value 

1000 

7.  Threshold 

1 

3.3Differences  in  Perspectives  of  Measures 
There  are  clearly  differences  in  the  perspec¬ 
tive  of  the  researchers,  analysts,  testers  and 
trainers.  This  is  due  in  part  to  the  differenc¬ 
es  m  disciplines,  but  it  can  also  be  attributed 
to  the  different  stages  of  acquisition  in 
which  communities  interact  with  the  C^IEW 
system.  Researchers  typically  function  in  the 
long  term,  developing  several  new  technolo¬ 
gies  for  later  application  in  a  system.  Devel¬ 
opers  are  in  the  mid  to  long  term  in  devel¬ 
oping  requirements  of  a  few  potential  sys¬ 
tems.  Testers  perform  their  evaluations  of 
one  or  two  systems  in  the  near  to  mid  term 
as  a  system  reaches  its  later  milestones  prior 
to  fielding.  Trainers  operate  with  one  system 
in  the  near  term  when  the  fielding  com¬ 
mences.  Analysts  operate  across  the  entire 
range  working  with  researchers  to  imple¬ 
ment  new  technologies  in  analytic  models, 
supporting  the  developer  by  analyzing  sever- 
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al  concepts,  supporting  the  acquisition  pro¬ 
cess  in  general  with  COEAs  and  weapon 
system  versus  C^I  system  tradeoff  analyses. 

The  difference  in  time  horizon  and  the 
separation  of  the  communities  make  it  diffi¬ 
cult  to  coordinate  the  development  of  mea¬ 
sures.  However,  it  is  not  clear  that  this  must 
be  done  except  for  neatness.  The  EEA  are 
not  necessarily  the  same  for  each  communi¬ 
ty.  That  is  not  to  say  that  it  would  not  be 
valuable  for  each  community  to  pay  atten¬ 
tion  to  what  measures  are  used  by  the  oth¬ 
ers.  With  the  trend  toward  compressing 
these  timelines,  the  need  for  cooperation 
among  the  communities  appears  to  be  grow¬ 
ing. 

There  are  three  points  that  summarize 
the  concerns  of  the  workshop: 

•  The  COEA  process  is  putting  pressure 
on  early  identification  of  measures  and 
linkage  of  analysis  measures  to  testing 
measures. 

•  Measures  for  COEAs  must  enable  trade¬ 
offs  to  be  made  between  C^IEW  systems 
and  weapon  systems,  as  a  matter  of 
policy.  In  other  words,  MOFE  are  nec¬ 
essary.  But  MOFE  are  inherently  less 
discriminating  than  MOEs  or  MOPs. 

•  Existing  and  force-level  models 
should  be  used  to  avoid  the  necessity  of 
building  new,  unique  data  bases.  (This 
point  was  contested  later  by  the  sponsors 
of  the  meeting  who  assured  participants 
that  resources  would  be  made  available 
for  the  necessary  new  models  and  data.) 

3.4  Methodologies  for  Deriving  Mea¬ 
sures 

The  most  familiar  methodology  in  the 


MORS  community  is  the  MCES.  It  is  based 
on  the  scientific  method  and  the  cybernetic 
loop  model  of  C^.  Although  implementation 
is  not  complete,  it  is  difficult  to  attack  as 
unsound. 

A  more  mathematically  rigorous  method¬ 
ology,  an  offshoot  of  the  MCES,  has  been 
developed  by  Paul  Girard,  one  of  the  MCES 
originators.  The  latest  version  of  this  ap¬ 
proach  introduces  an  expected  utility  frame¬ 
work  to  deal  with  decision-making.  It  has 
not  yet  been  applied  to  a  real  problem  be¬ 
cause  of  implementation  difficulties. 

Another  offshoot  of  MCES  has  been 
proposed  by  several  TRAC  analysts.  A  read- 
ahead  paper  by  Mr  Mike  Ingram  and  Cap¬ 
tain  Robert  Short  proposed  a  functional 
breakout  based  on  the  Army’s  doctrine  as 
described  in  FM  100-5.  Their  methodology, 
called  Structured  Resolution  Approach, 
describes  three  levels  for  analysis  of  com¬ 
mand  and  control  systems.  A  mission  level 
examines  measures  that  determine  if  a  force 
was  able  to  accomplish  its  mission.  This 
level  generally  focuses  on  command  and 
control  measures  of  effectiveness.  An 
airland  battle  tenet  level  examines  the  ability 
of  a  C^  system  to  support  the  four  tenets 
(agility,  depth,  initiative,  synchronization). 
This  level  focuses  on  command  and  control 
measures  of  effectiveness.  Finally  an  imper¬ 
ative  level  focuses  on  the  ability  of  a  C^ 
system  to  contribute  to  implementing  the 
imperatives  of  AirLand  Battle  doctrine 
(unity  of  effort,  anticipation,  concentration, 
main  effort,  press  the  fight,  move  fast,  use 
terrain,  conserve  forces,  joint  and  combined 
arms,  morale).  This  level  relates  to  the 
MOE/MOP  for  systems.  The  Structured 
Resolution  Approach  shows  promise  for 
Army  C^IEW  analysis,  but  may  not  apply  to 
other  services. 
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This  approach  has  recently  been  modi¬ 
fied  to  examine  C^IEW  systems  in  terms  of 
their  contributions  to  the  battle.  Under  this 
modification,  C^IEW  systems  are  examined 
to  determine  their  ability  to  contribute  in 
various  areas  where  a  C^IEW  system  is 
expected  to  enhance  the  commander’s  execu¬ 
tion  of  the  battle.  Within  each  area  of  con¬ 
tribution,  measures  are  identified  which  de¬ 
scribe  the  amount  of  contribution  being 
made  by  the  C^IEW  system.  Example  areas 
of  contribution  are  shown  in  Table  in-2. 

Another  methodology  briefly  discussed 
was  the  Headquarters  Effectiveness  Tool 
(HEAT)  which  is  now  in  use  in  the  Army  as 
a  part  of  the  Army  Command  and  Control 
Evaluation  system  (ACCES)  for  exercise 
evaluation.  ACCES  has  a  cybernetic  loop 
like  the  functional  process  loop  in  Module  3 
of  the  MCES.  It  has  an  overall  MOE  of  ‘life 
of  the  plan’  and  a  set  of  MOE/MOPs  at 
each  stop  in  the  cycle.  They  compare  ob¬ 


servable  performance  with  the  standards 
derived  from  standard  operating  procedures 
(SOPs).  HEAT  has  a  proven  data  collection 
scheme  for  headquarters  processes  in  field 
exercises  or  experiments. 

A  final  methodology  is  the  new  frame¬ 
work  for  C^  within  the  Combat  Science 
definitions  devised  by  a  group  of  prominent 
analysts.  The  C^  definitions  have  been  sug¬ 
gested  by  Professor  Wayne  Hughes  in  his 
paper  Cf  in  a  Theory  of  Combat  Science 
(NPS  Technical  Report  No.  55-89-05).  This 
paper  draws  heavily  on  the  MCES  and 
HEAT  methodologies,  but  is  consistent  with 
the  broader  scope  of  Combat  Science. 

Of  the  methodologies,  to  date,  the  re¬ 
view  of  literature  and  analysts’  experience 
suggests  that  the  MCES  had  the  widest  sup¬ 
port  as  the  broadest,  most  applicable  (but 
incomplete)  methodology. 


Table  III-2.  Areas  of  Contribution  for  a  C^IEW  System 

•  Develop  situation  awareness 

•  Provide  a  common  picture  of  the  battlefield 

•  Facilitate  control  of  battle  tempo 

•  Reduce  shooter  to  shooter  timeline 

•  Filter  information 

•  Allow  anticipatory  actions 

•  Enhance  commander's  agility 

•  Support  synchronization  of  the  battle 

•  Support  commander's  initiative 

•  Provide  near  real-time  intelligence 
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Chapter  IV 


Quantifying  C^IEW  Measures 


4.0  Current  Capabilities 

There  are  three  current  efforts  within  the 
Army  to  produce  quantification  of  C^IEW 
measures.  The  first  is  the  Command  and 
Control  Functional  Area  Model  (C^FAM) 
effort  to  support  the  TRAC  command  and 
control  stupes.  It  incorporates  a  Model- 
Test-Model  approach  and  relies  on  exercises 
and  subject  matter  expert  (SME)  panels  for 
C^  input  data.  SME  panels  use  a  structured 
Delphi  technique  to  capture  data. 

The  C^FAM  approach  uses  command 
and  control  performance  models  linked  to 
force  effectiveness  models  to  determine  the 
effectiveness  of  a  system.  The  perfor¬ 
mance  models  are  Petri  net  representations 
of  information  flow  within  the  structure. 
The  goal  of  this  modeling  effort  is  to  repre¬ 
sent  all  battlefield  functional  areas  (BFA) 
from  theater  to  brigade  level.  The  BFAs 


include  maneuver,  fire  support,  intelligence, 
air  defense  and  combat  service  support. 
These  performance  models  are  then  linked 
to  an  appropriate  force  effectiveness  model 
such  as  Vector-m-Commander  (VIC),  Eagle, 
or  TACWAR.  This  linkage  provides  the  key 
to  demonstrating  the  combat  effectiveness  of 
the  system.  By  using  the  output  from  the 
performance  models  as  input  to  the  force 
effectiveness  models,  the  system’s  impact 
on  battle  outcome  can  be  determined.  This 
methodology  allows  the  conduct  of  a  perfor¬ 
mance  analysis  as  well  as  an  effectiveness 
analysis  for  the  system.  Figure  IV- 1 
provides  a  graphical  representation  of  the 
C^FAM  methodology. 

As  seen  in  the  figure,  the  methodology 
incorporates  a  modeling  tool  called  Modeler 
to  develop  the  performance  models. 
Modeler  dlows  the  rapid  creation  of  Petri 


Figure  IV-1.  C^FAM  Methodology 
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_ Table  iV-1.  OPVIEW  Methodology _ 

OPVIEW 

(Operational  Value  of  Intelligence  and  Electronic  Warfare) 

•  OPVIEW  focuses  on  MOEs  for  Intelligence  (and  EW  somewhat)  as  they  relate  to  the  command  and 
control  decision  process. 

•  More  specifically,  the  emphasis  is  on  mission  accomplishment,  support  of  the  plan  and  monitoring  the 
plan  through  PIRs  (prioritized  intelligence  requirements). 

•  The  methodology  is  based  on  PIRs. 

•  Any  PIR  can  be  classified  as  a  combination  of  eight  functions: 

■  Detect 

■  Locate  Generally 

■  Locate  Precisely 

■  Classify 

■  Identify 

■  Trade 

■  Acquire 

■  BDA  (assess  remaining  operational  capability) 
o  Intelligence  collection  assets  attempt  to  meet  PIRs. 

o  Meeting  (or  failing  to  meet)  PIRs  in  a  timely  manner  provides  a  perception  to  the  command  element, 
o  The  perception  provides  information  to  the  plan  status/accomplishment, 
o  Plan  status  affects  the  decision  process, 
o  The  decisions  affect  the  forces, 
o  Orders  to  the  forces  affect  the  battle  outcome. 

•  EW  supports  these  eight  PIR  functions,  and  counters  the  enemy's  ability  to  support  his  own  eight  PIR 
functions. 

•  This  methodology  has  been  applied  to  noncombat  as  well  as  combat  missions. 

•  The  methodology  has  been  applied  in  three  model  applications. 


nets  to  represent  the  information  processing 
in  a  command  and  control  structure.  Mod¬ 
eler  was  developed  by  Alphatech,  Inc.  for 
the  Air  Force,  but  it  is  now  being  used  by 
numerous  military  and  civilian  organiza¬ 
tions. 

The  second  approach  is  employed  at  the 
ATCCS  Experiment  Site  (AES)  located  at 
Fort  Lewis,  Washington.  In  this  approach 
the  activities  of  the  9th  Infantry  Division 
(ID)  headquarters  have  been  extensively  re¬ 


corded  and  analyzed  in  field  exercises.  In 
this  effort  there  are  a  number  of  headquar¬ 
ters  tents  and  shelters  which  are  instrument¬ 
ed  with  TV -recorders.  Additionally,  a  data 
capturing  system  taps  into  all  data  bases  in 
the  headquarters.  This  effort  has  developed 
a  cadre  of  experienced  analysts  with  an 
understanding  of  the  process  flow  within  a 
command  post.  Unfortunately,  the  9th  ED 
has  been  deactivated,  but  part  of  the  7th  ID 
will  move  to  Fort  Lewis  and  the  AES  will 
probably  continue. 
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A  third  effort  is  the  RAND  Operational 
Value  of  Intelligence  and  Electronic  Warfare 
(OPVIEW)  methodology.  This  process  is 
now  in  development  as  three  related  models 
which  are  based  on  the  human  translation  of 
Prioritized  Intelligence  Requirements  (PIRs) 
into  intelligence  taskings.  The  results  of 
collection  activities  are  ^splayed  in  a  series 
of  simple  plots.  Table  IV- 1  describes  the 
approach  in  some  detail.  It  is  an  aggregate- 
level  approach  which  may  not  have  suffi¬ 
cient  detail  to  rqjresent  C^IEW. 

4.1  Problems  with  Current  Capabilities 

The  existing  capabilities  have  several 
major  problems.  In  developing  a  capability 
to  measure  C^iBW,  it  is  difficult  to  identify 
and  structure  the  decisions  involved  in  or 
impacting  on  force-level  campaigns.  The 
scope  and  amount  of  detail  required  in  the 
various  headquarters  areas  (intelligence, 
operations,  logistics,  and  EW  in  addition  to 
other  supporting  services)  is  not  clear.  There 
is  no  complete  doctrinal  set  of  the  decisions 
at  the  joint  level.  As  LTC  Shoffner  said  in 
his  presentation  (see  page  31),  the  US  ap¬ 
proach  to  command  is  more  parallel  and 
intuitive  than  the  Soviet  cybernetic  model. 
Service  SOPs  focus  on  procedures  for  plan¬ 
ning  rather  than  control  decisions. 

Modeling  of  information  flows  to  sup¬ 
port  decisions  is  weak.  is  often  modeled 
by  message  traffic  or  message  size  (bytes) 
alone.  While  this  type  of  modeling  is  good 
for  communications  capacity  analysis,  this 
approach  cannot  measure  the  content  of  the 
messages  nor  directly  determine  changes  on 
the  basis  of  the  information.  The  informa¬ 
tion  requirements  of  some  decisions  are  not 
easily  established,  in  particular,  the  more 
creative  command  decisions. 

There  is  a  poor  match  of  data  required 


within  our  models.  Values  for  performance 
characteristics  that  are  required  for  direct 
measurement  or  in  models  to  produce  effec¬ 
tiveness  measures  are  very  hard  to  obtain 
under  the  conditions  in  which  they  are 
collected/  evaluated. 

There  is  a  limited  capability  to  collect 
and  quantify  C^IEW  measures  at  the  cam¬ 
paign  level.  Collection  opportunities  are 
limited  simply  because  of  the  number  of 
exercises  that  involve  the  campaign  level. 
When  exercises  are  conducted,  there  ia  an 
inherent  difficulty  in  seeing  real  C^IEW, 
because  the  exercises  tend  to  be  scripted, 
designed  for  training  etc. 

There  are  several  problems  associated 
with  the  data  collection  process.  One  is  that 
there  is  often  limited  access  to  tests  and 
exercises.  Units  conduct  tests  and  exercises 
for  purposes  that  are  often  at  odds  with  the 
data  collection  process.  Then  there  is  a  com¬ 
mon  purpose,  unit  commanders  often  do  not 
want  additional  ‘evaluators’  in  the  area  for 
fear  of  having  their  units  judged  by  another 
set  of  eyes.  Even  if  data  were  to  be  collect¬ 
ed,  it  would  consist  primarily  of  what  was 
done  with  perhaps  some  insight  as  to  why. 
It  would  require  much  more  extensive  in¬ 
quires  (and  be  more  intrusive  to  exercises) 
to  determine  what  potential  options  were 
considered  but  rejected. 

Data  collection  is  currently  performed  on 
an  ad  hoc  basis.  In  most  cases  the  collection 
process  uses  subjective  methods,  nominal 
sources  and  focuses  on  tests  and  exercises 
under  some  artificial  conditions. 

Another  problem  is  the  lack  of  collection 
resources.  Other  than  organizations  such  as 
AES,  which  is  formally  structured  to  collect 
data,  most  analytic  agencies  must  depend  on 
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internal  resources  to  collect  C^IEW  data. 
Data  reduction  would  also  be  very  manpow¬ 
er  intensive. 

4.2  Solution  to  Problems 

There  are  some  solutions  to  the  problems 
mentioned  above.  First,  continued  automa¬ 
tion  of  training  and  collection  devices  should 
improve  the  data  collection  capability  and 
our  understanding  of  what  to  collect.  Sec¬ 
ond,  there  must  be  more  top  down  incen¬ 
tives  for  organizations  to  cooperate  and 
support  a  data  collection  effort.  Finally,  the 
communities  and  services  need  to  develop 
common  data  stmctures  for  C^EEW  that  will 
meet  the  needs  of  all. 

There  is  clearly  a  need  for  research, 
developer,  analysis,  training  and  testing 


communities  to  work  together  to  resolve 
some  of  the  issues  related  to  C^IEW  mea¬ 
sures.  As  funding  and  resources  become 
scarcer,  the  opportunities  to  conduct  individ¬ 
ual  community  data  collection  efforts  will 
disappear.  It  is  imperative  that  all  communi¬ 
ties  are  allowed  to  participate  in  every  data 
collection  opportunity  that  occurs.  This 
means  that  the  organizations  that  control  a 
data  collection  opportunity  must  have  the 
proper  incentive  to  allow  outside  organiza¬ 
tions  from  other  communities  to  participate. 
Training  units  must  have  a  good  reason  to 
allow  additional  observers  into  their  training 
exercises.  Interservice  data  collection  and 
evaluation  of  same  will  undoubtedly  open  aU 
kinds  of  sensitivities. 
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Chapter  V 

Tools  and  Methodologies  for  C^BEW  Evaluation 


5.0  Deficiencies  in  Tools  and  Methodol¬ 

ogies 

The  deficiencies  associated  with  our 
current  tools  and  methodologies  are  signifi¬ 
cant.  Our  current  tools  are  too  manpower 
intensive.  We  must  rely  on  a  multitude  of 
observer/controllers  to  perform  the  data 
collection  for  C^IEW  evaluation.  The  nature 
of  the  command  and  control  process  limits 
our  ability  to  use  automated  data  collection 
for  every  aspect  of  the  process. 

Although  we  must  perform  C^EEW  force- 
level  analyses  to  show  the  effect  of  C^IEW 
systems  on  the  outcome  of  battle,  our  force- 
on-force  models  that  are  supposed  to  support 
these  analyses  are  limited  in  their  explicit 
representation  of  the  process.  Most  of 
these  models  rely  on  a  man-in-the-loop  to 
capture  the  initiative  aspect  of  the  command¬ 
er  and  to  perform  the  command  and  control 
process.  Adding  the  man-in-the-loop  to 
simulations  essentially  nullifies  any  ability  to 
replicate  a  run  and  it  normally  lengthens  the 
run  time. 

Scenario  dependence  in  these  simulations 
is  very  high.  In  most  cases,  the  scenarios 
are  developed  for  weapon  system  evaluation 
rather  than  C^IEW  system  evaluation.  The 
result  is  that  C^IEW  systems  do  not  get 
stressed  during  the  battle  and  become  sec¬ 
ondary  factors  in  determining  the  outcome 
of  the  battle. 

Also,  the  technology  to  move  from 
system  characteristics  to  force-level  results 
is  very  limited.  We  have  made  only  limited 


progress  in  linking  the  performance  charac¬ 
teristics  of  a  C^IEW  system  to  battle  out¬ 
come.  Capacity  bottlenecks  are  often  not 
modeled  in  higher  level  models. 

The  process  in  campaign  level  models 
may  not  be  adequate  for  the  entire  purposes 
of  all  communities.  There  is  a  need  for 
linkage  of  simplified  processes  in  campaign 
models  to  more  detailed  examination  in  finer 
granularity  models. 

In  addition,  there  are  deficiencies  due  to 
a  lack  of  coordination  across  services  and 
agencies  as  well  as  resource  constraints.  If 
we  have  C^IEW  interoperability  problems  in 
the  real  world,  can  we  reaUy  expect  our 
models  to  behave  any  better? 

5.1  Overcoming  Current  Deficiencies 

We  should  perhaps  e5q)lore  new  theories 
of  combat  and  the  supporting  processes, 
i.e. ,  in  combat  science  stochastic  decision 
models.  This  includes  applying  of  other 
methodologies,  such  as  chaos  dynamics  and 
fuzzy  logic  to  the  C^IEW  problem. 

Several  changes  are  needed  to  place 
C^IEW  systems  on  a  comparable  level  with 
weapons  systems.  Scenarios  and  models 
need  to  be  developed  or  extended  to  specifi¬ 
cally  support  C^IEW  evaluation.  Policy 
makers  in  DoD  must  re-examine  current 
acquisition  requirements  to  determine  if 
changes  are  need  to  accommodate  the  differ¬ 
ent  nature  of  C^IEW  systems.  There  needs 
to  be  a  better  link  between  the  Mission 
Needs  Statement  (MNS),  Operational  Re- 
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quirements  Document  (ORD),  COEA,  and 
the  Test  and  Evaluation  Master  Plan 
(TEMP). 

We  should  strengthen  our  joint  efforts 
such  as  the  National  Training  Center,  Na¬ 
tional  Test  Bed,  and  Defense  Modeling 
Simulation  Office  (DMSO).  We  need  to  be 
able  to  respond  to  pressure  for  standardiza¬ 
tion,  verification,  and  validation  procedures 
from  Congress,  GAO,  and  users.  We  need 
to  explore  new  computer  tools  such  as 
object-oriented  models,  simulation  languages 
with  libraries  of  objects,  and  menu-driven 
model  building  similar  to  the  CAD/CAM 
process.  Finally,  we  may  need  to  develop 
new  models  of  C^,  perhaps  in  the  form  of 
Petri  nets,  conditional  possibility,  and  vari¬ 
able  resolution  modeling  and  use  these  as 
providers  of  data  to  our  more  traditional 
attrition  process  models. 

There  are  several  opportunities  that  need 
to  be  explored  as  sources  of  data  to  support 
models.  These  include: 

1.  Leavenworth  training  -  Battle  Com¬ 
mand  Training  Program  (BCTP)  and  Tacti¬ 
cal  Commander  Development  Course 
(TCDC),  and  senior  leadership  conferences 
with  1-4  star  generals. 

2.  National  Training  Center  -  ARI, 
Monterey  has  permanent  data  (video,  graph¬ 
ics,  audio,  and  hard  copy). 

3.  ATCCS  Experimentation  Site  -  AES 
maintains  complete  video  documentation  of 
HQ  exercises. 

4.  Battle  Force  In-Port  Training  (BFIT) 
-  Battle  group  level,  simulation  driven  with 
HEAT  data  collection. 


5.  FLAG  Air  Force  C^  exercises  with 
some  linkage. 

6.  Wargaming  at  service  schools. 

7.  Future  -  Embedded  training  in  every 
weapon  system  now  required.  This  wiU 
require  a  new  community  of  officers  with 
expertise  in  simulation. 

5.2.  Improving  Our  Tools  and  Method¬ 

ologies 

The  working  groups  discussed  a  wide 
range  of  possibilities,  but  had  limited  suc¬ 
cess  with  obtaining  closure,  except  concern¬ 
ing  the  desirability  of  the  Model-Test-Model 
approach.  Five  recommendations  are: 

1.  Develop  a  mechanism  for  coordina¬ 
tion  among  the  various  communities  and 
services. 

2.  We  need  an  orchestrated  set  of  en¬ 
hanced  tools  to  meet  the  C^IEW  evaluation 
needs.  This  includes: 

•  a  means  to  elicit  expert  opinion; 

•  constructive,  virtual  models  and 
simulations  that  treat  explicitly; 

•  participation  m  live  exercises,  e.g., 
NTC. 

3.  Use  the  Model-Test-Model  approach. 

4.  Develop  a  special  proposal  for  ad¬ 
dressing  a  specific  system  —  a  joint  system 
is  preferred. 

5.  We  need  a  vehicle  to  disseminate  our 
progress. 
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Chapter  VI 


Findings  and  Recommendations 


6.0  Findings 

The  workshop  produced  several  major 
findings.  Although  most  of  these  do  not 
address  the  original  questions  associated 
with  the  objectives,  the  findings  do  highlight 
the  fact  that  there  is  still  a  long  way  to  go  to 
integrate  the  efforts  of  the  various  commu¬ 
nities  to  allow  them  to  work  together  to 
achieve  mutually  supporting  objectives. 
These  findings  also  reveal  the  need  to  revi¬ 
talize  the  work  in  this  field  to  provide  a 
common,  acceptable  terminology  across 
services  and  communities.  The  findings  can 
be  categorized  into  five  areas:  processes, 
measures  of  merit,  methodologies,  tools  and 
community  communications. 

Processes 

The  modeling  efforts  need  to  focus  on 
the  C^I  processes  that  can  adequately  be 
represented. 

The  C^I  process  can  be  decomposed  into 
two  parts:  automatable  and 

non-automatable.  Automatable  processes 
include  information  processing,  situation 
development,  control,  communications,  and 
concept  evaluation.  Non-automatable  pro¬ 
cesses  include  concept  formulation,  building 
and  updating  perceptual  models.  We  have 
the  capability  to  represent  automatable 
processes  m  our  models,  but  we  must  cur¬ 
rently  rely  on  human  intervention  to  repre¬ 
sent  and  perform  non-automatable  processes. 

The  impact  of  £W  must  be  considered 
in  analyses  and  concepts  of  operations. 

We  saw  clearly  in  Desert  Storm  the 
effect  of  EW  on  the  success  of  our  opera¬ 
tion.  It  was  a  significant  combat  multiplier 


when  used  properly  in  the  concept  of  the 
operation.  However,  for  the  most  part,  the 
overall,  joint  effect  of  EW  on  force  level 
operations  has  been  ignored  in  our 
force-on-force  models.  We  need  to  improve 
our  representation  of  EW  to  show  its  impact 
on  shortening  the  planning  cycle  time  and 
adding  to  the  confusion  on  the  battlefield  as 
it  degrades  the  opposing  commander’s  situa¬ 
tion  awareness. 

Measures  of  Merit 

Note:  The  term  Measures  of  Merit 
(MoMs)  was  introduced  to  subsume  all  of 
the  measures  that  characterize  a  C^IEW 
system’s  performance  (i.e..  Measures  of 
Performance),  its  functional  effectiveness 
(i.e..  Measures  of  Effectiveness),  and  its 
impact  on  force  effectiveness  (i.e. ,  Measures 
of  Force  Effectiveness). 

Measures  of  Merit  are  defined  by  the 
level  of  analysis  and  the  context  in  which 
they  are  measured. 

The  context  in  which  we  are  measuring 
MoMs  affects  the  way  in  which  we  define 
them.  Depending  on  our  perspective,  a 
measure  might  be  a  measure  of  performance 
or  a  measure  of  effectiveness.  It  depends  on 
the  question  being  answered  in  the  analysis. 

MoMs  have  the  attributes  of  a  name, 
category,  system  reference  (boundary),  a 
function  reference  (purpose),  units  of 
measure,  value  measured,  and  threshold 
value  (goal). 

Although  this  might  not  be  an  exhaustive 
set  of  attributes,  it  is  a  good  start  in  the 
process  of  defining  and  describing  MoMs. 
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Additionally  work  is  required  to  refine  this 
list  further. 

There  are  some  major  MOEs  for 
C^IEW  systems  that  should  be  considered 
in  evaluating  the  system. 

Based  on  our  experience  with  C^IEW 
systems  in  using  them  in  a  variety  of  situa¬ 
tions,  we  can  identify  some  major  MOEs 
that  should  be  considered.  These  include  the 
ability  to:  maintain  multiple  views  of  ene¬ 
my  courses  of  action;  formulate  and  evaluate 
multiple  friendly  courses  of  action;  config¬ 
ure  and  reconfigure  rapidly  for  new  situa¬ 
tions  (flexibility);  identify  and  assess 
‘ground  truth’  and  infer  from  it;  and  be  easy 
to  train  on  and  use. 

Alternative  C^IEW  systems  manifest 
greater  changes  in  MOPs  than  Measures 
of  Force  Effectiveness  (MOFE). 

We  should  expect  that  C^IEW  systems 
have  a  small  effect  on  the  overall  force 
effectiveness.  That  is,  it  is  not  reasonable  to 
say  that  a  change  in  one  ratio  will  win  the 
war.  We  need  to  be  careful  and  clever  in  the 
way  we  evaluate  the  effect  of  C^IEW  sys¬ 
tems  so  that  we  can  discriminate  between 
the  effects  of  a  C^I  system  and  other  factors 
to  properly  reflect  the  impact  of  a  C^I  sys¬ 
tem  on  force  effectiveness.  However,  tradi¬ 
tional  MOFEs  at  the  campaign  level  should 
not  be  rejected  as  a  subset  of  those  useful  in 
C^IEW  analyses,  especially  when  comparing 
sensors  and  shooters. 

Methodologies 

There  exist  some  promising,  but  in¬ 
complete  methodologies  to  deHne,  derive, 
and  measure  MoMs. 

We  care  a  great  deal  about  measures,  but 
we  still  need  an  underlying  methodology  to 
provide  a  logical  structure  for  how  we  ap¬ 
proach  this  very  complex  problem  to  gather 


these  measures.  A  lot  of  good  work  has 
been  accomplished  in  previous  efforts  on 
MOEs,  but  there  is  still  no  complete  or 
comprehensive  methodology  that  exists. 
However,  persistence  in  linking  MOPs  to 
MOEs  to  MOFEs  will  help  in  this  regard. 

Tools 

Existing  force-on-force  models  do  not 
contain  modules  that  adequately  represent 

processes. 

Current  force-on-force  models  have  not 
been  very  useful  for  analyzing  the  full  spec¬ 
trum  of  C^IEW  systems,  because  they  do  not 
explicitly  represent  command  and  control. 
This  limits  the  Model-Test-Model  paradigm 
that  is  needed  to  integrate  modeling  into  the 
test  and  evaluation  world.  Likewise,  the 
theory  of  how  to  smartly  aggregate  lower 
level  cause  to  aggregate  level  effect  needs  to 
be  extended. 

An  orchestrated  set  of  enhanced  tools 
is  required  to  meet  the  C^HSW  evaluation 
needs. 

There  is  no  one  magic  tool  that  meets 
our  needs  for  evaluating  C^IEW  systems. 
We  need  a  set  of  tools  that  will  allow  us  to 
capture  every  aspect  of  the  C^BEW  process 
to  properly  analyze  a  system.  These  tools 
must  include  the  means  to  elicit  expert 
opinions;  the  means  to  capture  activity  in 
live  exercises  such  as  at  the  National  Train¬ 
ing  Center  (NTC);  and,  constructive  and 
virtual  modeling  and  simulation  that  treat  C^ 
explicitly. 

Community  -Communications 

Continuous  interrelationships  are 
needed  among  the  key  communities  (de¬ 
velopers,  testers,  trainers,  and  re¬ 
searchers). 

We  have  learned  that  people  in  these 
different  communities  speak  different  lan- 
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guages  and  it  is  sometimes  difficult  to  com¬ 
municate  the  needs  of  one  community  to 
another.  We  need  to  work  more  closely 
together  to  find  the  common  ground  and 
make  our  efforts  mutually  supporting.  This 
is  true  not  only  within  a  specific  service,  but 
across  services. 

Security  classification  barriers  need  to 
be  overcome. 

Historically,  we  have  not  done  well  in 
playing  Sensitive  Compartmented  Informa¬ 
tion  (SCI)  systems  in  our  models  and  simu¬ 
lations.  Very  few  people  have  access  to 
these  systems  and  information  about  them. 
If  we  are  to  properly  represent  their  role  in 
the  battle,  we  need  to  overcome  some  of  the 
barriers  that  we  now  face. 

In  the  context  of  a  training  exercise, 
training  and  system  evaluation  goals  are 
frequently  at  odds  with  each  other. 

In  training  exercises,  we  have  very 
attractive  activities  with  large  groups  of 
people  gathered  together,  and  we  are  unable 
to  t^e  full  advantage  of  an  excellent  data 
collection  opportunity.  Although  system 
evaluators  see  this  opportunity,  those  being 
trained  naturally  view  the  evaluators  as 
additional  burdens  and  hindrances  to  their 
training.  It  is  also  difficult  to  isolate  system 
effects  from  human  training  effects  without 
the  benefit  of  multiple,  independent  repli¬ 
cations. 

6.1  Recommendations 

Several  recommendations  emerged  from 
the  Workshop. 

Clarify  the  different  levels  of  MoMs. 

More  work  is  needed  in  this  area  to 
accurately  define  the  levels  of  measures  to 
be  used  by  all  communities.  One  possible 
way  of  accomplishing  this  is  through  case 


studies  of  real  systems.  Particular  emphasis 
should  be  given  to  a  system  which  has 
potential  impacts  across  all  services,  e.g., 
JSTARS. 

Assess  the  feasibility  of  existing  meth¬ 
odologies  and  establish  a  plan  to  make 
them  more  credible  and  easier  to  apply. 

It  is  clear  that  in  this  area  there  are  no 
cook  book  solutions.  We  need  to  resist  the 
urge  to  be  overly  mechanistic.  The  analysis 
of  C^IEW  systems  will  require  creative  and 
innovative  thought.  Any  approach  taken 
should  be  tailored  to  bridge  the  disciplinary 
gaps  between  the  operational  and  the  techni¬ 
cal  levels.  Single  point  solutions  are  inade¬ 
quate  for  this  type  of  analysis.  Sensitivity 
analyses  are  mandatory.  Documentation  of 
these  efforts  should  be  made  available 
throughout  the  DoD  analysis  community. 

Explore  the  potential  value  of  applying 
functional  decomposition  techniques  to  en¬ 
hance  evaluation. 

One  suggestion  is  to  take  a  dendritic  ap¬ 
proach  to  the  problem. 

Encourage  training  activities  to  collab¬ 
orate  early  in  the  planning  process  with 
T&E  to  enhance  dual  use. 

Take  early  steps  to  incorporate 
evaluation  requirements  into  DIS  and 
WARSIM  2000. 

There  is  an  excellent  opportunity  to 
affect  the  usefulness  of  training  simulations 
for  the  testing  and  analysis  communities.  We 
must  take  advantage  of  this  to  provide  some 
input  into  the  development  of  these  simula¬ 
tions. 

Continue  the  trend  to  explicitly  model 
C’l  processes  in  force-on-force  models. 

If  we  are  able  to  do  this,  it  will  lay  the 
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foundation  to  do  Model-Test-Model  work. 
In  parallel,  develop  robust,  good  enough 
representations  for  higher  level  simulations 
that  are  tuned  by  more  detailed  examina¬ 
tions. 

Develop  theories  of  complex  systems 
management  and  control. 

It  is  important  that  we  not  only  develop 
these  theories  but  we  must  also  be  able  to 
translate  them  into  practical  use.  Perhaps 
there  are  lessons  to  be  learned  here  from  the 
civilian  communities  or  international  com¬ 
munities. 

Develop  measurable  combat  require¬ 
ments,  objectives,  and  tasks  to  permit  the 
tradeoff  analyses  of  C^I  and  combat  sys¬ 
tems. 

Explore  potential  role  of  evolving  syn¬ 
thetic  environments  on  C^I  analyses. 

Make  the  workshop  results  and  DIS 
symposium  report  widely  and  quickly 
available. 

We  need  to  energize  the  entire  OR  com¬ 
munity  to  tackle  this  problem  and  make 
them  aware  of  the  progress  that  has  been 
achieved  so  far.  We  need  to  include  a  clear 
and  concise  description  of  Lx)uisiana  Maneu¬ 
vers  and  its  role  in  this  effort. 

Enhance  the  exchange  of  information 
on  prospects  and  work  in  progress  on 
evaluation. 


Focus  forthcoming  MORS  working 
groups  on  the  key  issue  of  representation 
of  in  analysis. 

6.2  Summary 

The  challenge  in  working  with  C^IEW 
MOEs  is  to  overcome  not  only  the  curse  of 
dimensionality  but  also  the  curse  of  ambigu¬ 
ity.  Care  must  be  taken  to  inteipret  analysis 
results  to  ensure  that  they  are  telling  us  what 
we  think  they  are  telling  us. 

The  workshop  renewed  an  interest  in  the 
problems  and  issues  related  to  C^IEW 
MOEs.  The  challenge  for  the  community  is 
to  continue  where  this  workshop  ended  and 
maintain  the  interest  that  was  demonstrated 
throughout  the  workshop.  We,  in  separate 
services  and  separate  communities,  must 
work  closer  together  to  maintain  a  common 
language,  to  mutually  support  one  another, 
and  to  share  our  knowledge  and  experience 
to  improve  our  capabilities  to  support 
C^IEW  systems  development,  testing,  and 
training. 

The  results  of  the  Workshop  has  inspired 
at  least  one  participant.  Mike  Sovereign  has 
suggested  that  NFS  host  a  foUow-on  work¬ 
shop  to  further  develop  the  joint  effort 
initiated  here. 
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Appendix  A 

Excerpts  from  Workshop  Dinner  Speech 


COL  James  McDonough 
Director,  School  of  Advanced  Military  Studies 
US  Army  Command  and  General  Staff  College 


.  .  .  My  topic  tonight  is  What  is  a  Cam¬ 
paign?  That  is  a  rather  imposing  subject. 
On  the  one  hand,  it  is  a  formidable  chal¬ 
lenge  to  speak  to  aU  of  you  ORSA  experts 
in  what  should  otherwise  be  a  light  and 
easy-going  after  dinner  speech.  ...  On  the 
other  hand,  it  is  intriguing  to  talk  to  you 
about  my  topic,  what  is  a  campaign.  .  .  . 
My  approach  wiU  be  to  talk  about  where  we 
in  the  Army  are  going  with  our  doctrine. 

My  credentials  for  doing  this  are  two¬ 
fold.  Number  one  is  that  I  am  located  at 
SAMS  (School  of  Advanced  Military  Stud¬ 
ies)  and  number  two,  I  had  some  responsi¬ 
bility  in  preparing  the  next  version  of  FM 
100-5,  Operations,  the  Army’s  keystone 
doctrine.  .  .  . 

...  I  want  to  relate  doctrine  to  ORSA 
because  that  is  what  you  do  and  that  is  the 
task  before  us.  From  whence  does  our 
doctrine  come?  That  is  the  beginning  of  my 
talk  tonight.  How  do  you  take  all  that  you 
know,  all  that  you  have  learned  in  life  and 
bring  it  all  together  to  write  a  doctrine  for 
warfare  in  the  21st  century.  What  follows, 
then,  as  a  preface  to  the  doctrine  is  the 
intellectual  odyssey  that  gets  you  there. 

Let  me  begin  with  my  own  beginnings  in 
this  process.  ...  To  me  the  big  event  in  my 
young  life,  the  recent  event,  was  World  War 
n.  It  was  my  focus  of  warriorship  as  a 
young  man.  My  father  was  a  veteran  of 


World  War  H.  .  .  .  I  admired  him,  wanted 
to  be  like  him.  And  I  watched  all  the  great 
Hollywood  war  movies  —  A  Walk  in  the 
Sun,  a  gripping  display  of  infantry  soldiers 
fighting  in  Italy;  Twelve  O’clock  High,  the 
application  of  aiipower  to  warfare  and  the 
brave  men  that  fought  the  Battle  of  Britain 
and  took  the  campaign  to  the  Germans,  Run 
Silent,  Run  Deep,  the  thrilling  undersea 
battle  taking  the  fight  to  the  Japaiiese.  From 
all  those  reflections,  I  drew  this  idea  of 
heroes.  We  won  our  war  because  we  had 
heroes.  .  .  . 

When  I  became  a  little  bit  more  mature, 
I  began  to  understand  that  leadership  had 
some  manifestation  on  how  to  use  those 
heroes.  The  great  names  of  my  era  started 
to  become  known  to  me:  Dwight  D.  Eisen¬ 
hower,  President  of  the  United  States,  Su¬ 
preme  Allied  Commander,  five  star  gener¬ 
al;  George  C.  Marshall  of  the  Marshall 
Plan,  architect  of  the  war  who  would  later 
become  Secretary  of  State  and  Secretary  of 
Defense.  Vinegar  Joe  Stilwell  walking  his 
way  out  of  Burma  and  telling  the  press  we 
took  a  licking,  we  have  got  to  figure  out 
what  went  wrong  and  go  back  and  return  the 
favor  —  and  we  did.  Omar  Bradley,  the 
soldier’s  general;  Hap  Arnold,  founder  of 
the  Air  Force;  and  BuU  Halsey,  terror  of  the 
Japanese,  Chester  W.  Nunitz  rises  to  the 
occasion,  wins  our  great  sea  battle  during 
the  dark  days  of  1942,  when  our  seapower 
seems  to  have  been  destroyed;  Douglas 
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MacArthur,  a  man  of  vision  and  a  perennial 
general.  The  thought  entered  my  mind  we 
won  because  we  had  the  right  leaders  not 
just  heroes  and  soldiers. 

A  short  time  later,  somewhere  in  there, 
I  remember  reading  a  story  about  a  baseball 
team  at  Fort  Benning  in  the  early  1930s. 
The  composition  of  that  team  was  remark¬ 
able.  There  was  George  C.  Marshall  in  the 
stands,  Patch  in  left  field,  Gerow  at  short 
stop,  Bradley  coaching,  Devers  at  first  base 
and  on  and  on  it  went.  On  that  team  of  nine 
players  practically  every  one  of  them  rose  to 
a  three  or  four  star  general  before  World 
War  n  was  over.  The  thought  occurred  to 
me  that  it  is  not  just  soldiers  and  leaders, 
but  that  it  was  the  richness  of  our  nation  in 
materiel  and  human  resources  that  brought 
us  victory.  .  .  . 

A  little  later  in  life,  my  youth  behind 
me,  I  went  to  MIT.  I  began  to  run  into 
some  of  the  defense  intellectuals  of  the  day. 
They  had  a  different  way  of  looking  at  it. 
Some  of  you  can,  I  am  sure,  identify  with 
them.  You  are  contemporaries  and  proteges 
of  them.  George  Rathgens  who  argued 
adamantly  that  an  air  defense  system,  the 
ABM  system,  would  not  be  at  all  effective 
in  defending  the  United  States  against  a 
strategic  nuclear  attack.  He  was  opposed  just 
as  adamantly  and  just  as  convincingly  by 
Albert  WoUstetter  and  the  difference  be¬ 
tween  them  was  really  an  assumption  of  the 
probability  of  launch  and  successful  separa¬ 
tion  of  the  warhead  from  the  booster  rocket. 
The  difference  between  their  original  as¬ 
sumptions  was  only  0.2  as  I  recall  it,  but  it 
made  all  the  difference  in  the  world  at  the 
far  end  as  to  whether  or  not  the  US  could 
successfully  be  defended  by  a  missile  sys¬ 
tem. 


William  Kaufmann,  one  of  the  original 
whiz  kids"  one  of  the  first  at  RAND  (and 
RAND  is  here  tonight),  taught  me  the  busi¬ 
ness  of  quantifying  military  resources  and 
calculating  their  military  effects  against 
likely  opponents.  His  studies  of  conventional 
and  nuclear  warfare  were  very  quantified, 
very  deep,  very  provocative.  These  men, 
and  others  like  them,  were  saying  there  are 
ways  to  think  about  all  this,  that  if  you 
study  the  piece-parts  that  can  be  applied, 
they  will  tell  you  how  to  put  together  the 
best  program,  the  best  expenditure  package, 
the  best  technology  to  ensure  victory  as  was 
planned.  The  scientists,  the  nuclear  theo¬ 
rists,  the  systems  analysts  were  aU  suggest¬ 
ing  a  method  of  determining  How  Much  Is 
Enough  in  military  preparedness. 

Herman  •  Kahn,  Charlie  Hatch,  Alain 
Enthoven  led  me  to  a  study  of  Robert 
McNamara  and  his  roots  in  World  War  H, 
how  he  applies  his  Harvard-learned  skills  of 
systems  analysis  to  the  8th  Air  Force  in 
Europe  and  later  on  to  the  20th  Air  Force  in 
the  Pacific,  then  on  to  the  Ford  Motor  Com¬ 
pany.  He  rises  to  be  the  Secretary  of  De¬ 
fense  and  he  institutes  the  PPBS  system  by 
which  we  fight  the  war  in  Vietnam  and 
structure  much  of  our  defense  business  after 
the  war,  a  system  based  on  the  belief  that 
you  can  look  with  precision  and  find  the 
critical  node,  analyze  it,  vary  it  and  measure 
the  outcomes.  P.  M.  S.  Blackett  is  his 
intellectual  godfather  whose  classic  studies 
of  war  talks  about  the  relationships  between 
tactics  and  strategy  and  uses  as  the  basis  for 
his  ideas  the  measurement  of  the  effective¬ 
ness  of  air  and  anti-submarine  warfare  in 
World  War  H,  and  how  best  to  apply  the 
resources  of  war.  My  store  of  information 
on  warfare  was  now  starting  to  get  quite 
large  and  unwieldy,  indeed. 
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Finally,  in  my  thinking  I  show  up  at 
SAMS,  encouraged  to  do  so  by  the  prescient 
Huba  Wass  de  Czege.  He,  as  you  know, 
was  so  instrumental  in  the  rediscovery  of 
operational  art  and  the  convincing  of  our  the 
Army  that  we  needed  to  work  on  that.  From 
him  and  through  SAMS,  I  begin  the  studies 
of  theories  of  war  that  run  the  gamut  from 
Clausewitz,  Jomini,  Sun  Tzu,  Mahan, 
Douhet,  Liddell  Hart  and  Van  Creveld,  and 
the  Russians  Svechin  and  Tukhachevsky.  At 
this  point,  one  starts  to  come  to  grips  with 
the  reality  that  war  is  a  very  complex  busi¬ 
ness,  that  after  all  the  study  of  war  is  just  a 
jumble,  just  like  war  itself;  but  out  of  it 
must  come  order.  You  have  to  study  the 
parts  but  integrate  it  into  the  whole. 

When  we  took  upon  ourselves  the  task  of 
writing  the  Army’s  doctrine  we  understood 
from  the  get-go  that  forecasting  the  way  of 
war  was  the  most  difficult  thing  in  front  of 
us,  that  whatever  doctrine  we  devised  would 
live  in  a  sea  of  uncertainty  —  the  uncertain¬ 
ty  of  whether  or  not  we  had  it  right,  and 
that  a  wrong  guess  could  result  in  calamity. 

Michael  Howard  at  Yale  advises  —  and 
we  took  his  words  to  heart  —  that  the  best 
you  can  hope  for  is  not  to  have  it  so  terribly 
wrong  that  you  lose  at  the  onset  of  the  first 
battle.  That’s  not  a  very  optimistic  view 
point.  As  we  began  the  process  of  redoing 
our  doctrine,  we  hoped  to  do  better.  Wheth¬ 
er  we  did  or  not  remains  to  be  seen. 

My  message  tonight  as  I  take  you  into 
the  business  of  doctrine  is  threefold:  the 
first  thing  is  to  recognize  that  warfare  — 
understanding  it,  planning  it  out  and  fighting 
it  victoriously  —  is  both  an  art  and  a  sci¬ 
ence;  the  second  thing  is  that  we  chose  to 
describe  how  we  as  an  Army  thought  about 
war,  not  prescribe  how  we  expected  every¬ 


one  to  fight  it  in  detail;  the  third  is  that  your 
particular  skills  of  operations  research  and 
systems  analysis  can  help  in  improving  our 
application  of  the  tools  of  war  in  specific 
instances,  but  that  even  then  it  remains  an 
uncertain  business,  affected  by  the  so  many 
intangibles  the  Clausewitz  defined  as  fog 
and  friction  and  chance. 

.  .  .  What  are  the  major  points  of  our 
operational  doctrine?  What  follows  is  just 
really  a  surface  treatment  of  the  major  is¬ 
sues. 

The  first  thing  is  the  idea  of  the  com¬ 
pression  of  the  strategic,  operational  and 
tactical.  Although  there  are  different  levels 
of  war,  the  line  between  them  are  fine  and 
not  all  that  clear.  Any  commander  must 
realize  that  he  transgresses  from  one  to  the 
other  in  short  order  if  the  conditions  are 
right.  This  has  major  ramifications  on  the 
way  we  fight.  Second,  we  need  to  define  the 
end  state  at  the  beginning  —  not  at  the  end. 
You  need  to  know  where  you  are  going 
before  you  start.  Third,  from  that,  comes 
the  understanding  that  the  campaign  doesn’t 
end  with  the  last  shot.  You  begin  a  cam¬ 
paign  in  order  to  accomplish  strategic  ends. 
Often  those  strategic  ends  are  not  really 
anchored  until  you  put  into  effect  operations 
that  wiU  continue  after  the  end  of  war. 
Fourth,  you  can  not  assume  yourself  at  the 
point  of  the  struggle  —  the  locale  of  the 
campaign  —  at  the  start.  That  was  a  conve¬ 
nience  of  our  doctrine  during  the  Cold  War. 
We  assumed  ourselves  present.  We  would 
fight  the  war  in  Europe  and  we  were  there 
in  force  at  the  start.  Forward  defenses  was 
our  strategy,  it  was  our  watch  word.  We 
had  over  300,000  Americans  in  uniform  m 
Europe,  even  before  any  fighting  began, 
matched  by  up  to  60  days  of  supply.  Force 
projection  is  the  watch  word  now  and  with 
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it  a  number  of  difficult  tasks.  We  will  have 
to  get  ourselves  to  the  contested  region, 
along  with  virtually  all  our  supplies.  That 
will  be  a  difficult  and  challenging  business. 
I  think  ORSA  could  very  definitely  apply 
itself  there.  How  do  you  get  there,  when,  in 
what  order,  and  for  what  effect. 

That  leads  quite  directly  to  the  next 
concern,  integration  of  all  types  of  war  into 
a  keystone  doctrine.  Not  just  a  conventional 
war,  but  all  kinds  of  war:  the  conventional, 
the  unconventional,  the  nuclear,  weapons  of 
mass  casualties  —  all  of  these  can  coexist 
simultaneously  in  the  same  theater  of  opera¬ 
tions.  One  can  flow  into  the  other  and  they 
must  aU  be  addressed  in  a  holistic  maimer. 
That  should  be  done  in  our  keystone  doc¬ 
trine. 

We  take  into  account  that  not  all  we  do 
is  warfighting,  that  we,  the  Army,  often  find 
ourselves  in  operations  other  than  war. 
These  operations  are  not  unimportant.  They 
do  not  degrade  our  central  focus  on 
warfighting.  In  fact,  they  are  related  to 
virtually  any  campaign  you  can  concoct.  But 
they  are  also  important  in  and  of  them¬ 
selves.  They  must  be  understood  and  the 
doctrine  must  be  there  to  help  us  think  about 
it. 

In  one  of  our  most  central  concerns  — 
and  I  think  innovative  thoughts  —  for  devis¬ 
ing  doctrine,  we  came  to  terms  with  under¬ 
standing  that  there  is  an  American  way  of 
war,  one  that  is  uniquely  American,  and 
that,  therefore,  both  constrains  us  and  brings 
us  the  opportunity  to  think  about  war  in 
certain  ways.  For  example,  one  outcome  of 
that  is  the  desire  to  achieve  quick,  decisive 
victory  with  minimal  casualties. 

We  related  this  idea  of  an  American  way 


of  war  to  a  world  of  tighter  communica¬ 
tions,  a  shrinking  globe  where  CNN  was 
ever  present.  We  must  imbed  in  the  mind  of 
the  commander  that  once  he  initiates  his 
operations,  the  sands  of  time  are  running 
through  the  hourglass;  he  is  being  observed, 
in  ways  that  are  unknown  to  him.  Whatever 
action  he  takes  will  have  political,  and  there¬ 
fore  strategic,  consequences  in  the  homes  of 
the  voting  public.  Consequently,  it  will  have 
an  impact  on  the  conduct  of  our  campaign. 

We  entered  into  at  least  the  first  draft 
with  the  idea  of  versatility  as  a  tenet,  adding 
to  the  original  four  of  initiative,  agility, 
depth,  and  synchronization.  It  was  not 
enough  to  be  agile,  to  shift  within  the  the¬ 
ater  of  operations  from  one  mode  of  fighting 
to  the  other.  We  also  had  to  be  able  to  shift 
quickly  between  theaters  and  between  types 
of  warfare  and  from  warfare  to  operations 
other  than  war.  We  recognized  that  we 
needed  to  do  all  this  all  with  a  shrinking 
base,  that  it  had  to  be  done  professionally 
and  proficiently,  without  ever  losing  stride. 
We  also  attempted  to  articulate  a  concept  we 
call  battle  space.  Battle  space,  yet  to  be 
finely  resolved,  deals  with  both  the  intellec¬ 
tual  concept  and  the  physical  space  in  which 
the  commander  fights  his  fight.  We  have 
likened  it  to  the  ring  mastery  of  the  master¬ 
ful  boxer  who  knows  within  the  confines  of 
his  ring  --  his  battle  space  —  he  must  oper¬ 
ate  by  using  every  available  asset.  He  must 
decide  when  to  strike,  when  to  maneuver, 
when  to  attack,  when  to  defend,  and  when 
to  put  in  the  finishing  blow. 

We  also  tried  to  come  to  grips  with  the 
multi-dimensionality  of  war.  We  no  longer 
call  it  just  AirLand  Battle;  in  fact  that  term 
has  faded.  We  have  not  walked  away  from 
airland  battle  but  we  recognize  warfare  as 
something  a  lot  bigger  than  that.  It  is 
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air-land-sea;  it  is  space;  it  is  special  opera¬ 
tions;  it  is  electromagnetic;  it  is  joint  and 
combined;  it  is  multi-dimensional  war  and  it 
must  all  be  brought  together  into  one  whole. 

Tempo  enters  into  the  con  of  our  doc¬ 
trine  and  what  we  mean  by  it,  which  is  not 
just  speed.  There  are  times  when  you  speed 
up  and  you  quicken  the  tempo;  there  are 
times  when  you  slow  down  and  you  back  off 
the  tempo.  Certainly  it  is  a  measure  in  the 
effectiveness  of  war  and  has  central  impor¬ 
tance  to  successful  warfighting.  We  took  an 
old  concept  —  deep,  close  and  rear  battle  — 
and  we  really  thought  about  that.  We  rede¬ 
fined  its  purpose.  We  used  to  say  that  deep 
battle  was  germane  only  so  far  as  its  effec¬ 
tiveness  to  close  battle.  Today  we  believe 
that  is  not  the  sole  determiner.  Deep  battle 
must  also  be  related  to  its  effect  on  future 
battles.  The  simultaneity  of  operations 
throughout  the  depth  of  the  battle  became  a 
more  powerful  concept.  It  is  geared  not  only 
to  close  battle  but  to  future  events  as  well 
that  determine  the  outcome  of  the  campaign. 


Finally,  in  this  short  list  of  major  points 
of  what  we  did  with  our  doctrine  we  tried  to 
relate  technology  to  doctrine.  Although  we 
did  not  use  words  as  high-falutin  as  this,  the 
idea  we  applied  is  that  there  is  a  symbiotic 
relationship  —  one  affected  the  other  —  and 
they  had  to  come  together  in  a  sensible  way. 
You  just  do  not  reach  for  whatever  technolo¬ 
gy  that  happens  to  be  out  there  and  overlay 
that  existing  doctrine.  You  just  do  not  read¬ 
just  the  doctrine  every  time  a  new  item 
comes  down  the  pike.  Both  had  to  adapt  to 
the  other  and  from  that  integrate  the  rela¬ 
tionship  between  technology  and  doctrine. 
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Appendix  B 

Excerpts  from  Keynote  Address 

LTG  Wilson  A.  Shoffner 

Commanding  General,  Combined  Arms  Command 
US  Army  Training  and  Doctrine  Command 


Our  Army  faces  many  challenges  as  we 
respond  to  a  rapidly  changing  world.  Per¬ 
haps  our  greatest  challenge  is  command  and 
control  and  tactical  decision  making.  My 
purpose  today  is  to  address  these  two  very 
important  topics  so  you  will  understand 
where  our  Army  is  going,  with  respect  to 
doctrine,  and  what  you  can  do  to  better 
prepare  our  trained  and  ready  Army. 

As  we  work  together  to  meet  these  chal¬ 
lenges,  it  is  helpful  to  understand  how  future 
battle  dynamics  —  how  soldiers  fight  on  the 
battlefield  —  reveal  our  requirements  for 
modernizing  the  force.  The  battlefield  has 
always  been  complicated.  The  future  battle¬ 
field  will  continue  to  be  complex. 

Battlefield  tempo  is  accelerating.  Sys¬ 
tems  are  more  lethal,  tactical  mobility  is 
increasing,  range  of  fires  is  increasing  and 
more  information  and  intelligence  systems 
are  available.  Together,  this  rapid  pace  will 
make  battles  short  and  violent.  Increased 
tempo  on  a  complex  and  lethal  battlefield 
has  several  implications  for  decision  makers. 
Information  may  be  ambiguous.  For  sure, 
decisionmaking  wiU  have  less  time  for  deci¬ 
sions  that  will  have  significant  consequenc¬ 
es.  This  battlefield  will  require  agile  leaders. 

The  future  requires  agUe  leaders  capable 
of  acting  faster  than  the  enemy  —  mentally 
and  physically.  Leaders  must  be  capable  of 


rapidly  formulating  concepts,  planning 
operations,  making  decisions  and  pressing 
the  fight.  Agile  leaders  who  quickly  and 
accurately  apply  the  conditions  of  their 
environment  will  produce  decisive  victories. 

Command  and  control  procedures  must 
support  the  agile  leaders  on  tomorrow’s 
battlefield.  We  must  be  capable  of  providing 
the  enemy  at  least  three  ways  to  die  simulta¬ 
neously.  That  means  effectively  synchroniz¬ 
ing  our  forces  and  placing  decisive  combat 
power  at  the  right  place  and  time.  This 
requires  simple  procedures,  unified  con¬ 
cepts,  nested  intents,  commanders  who  can 
see  and  understand  the  battlefield,  and  com¬ 
manders  capable  of  concurrently  considering 
current  and  future  operations.  Calm,  compe¬ 
tent,  confident  commanders  capable  of 
bringing  order  to  chaos  must  understand 
command  and  control. 

The  acronym  "C^"  seems  to  suggest  that 
command  and  control  is  one  word.  Com¬ 
mand  and  control  is  not  one  word.  Each  is 
different  and  carries  with  it  significantly 
different  responsibilities. 

Command  is  an  art.  Commanders  formu¬ 
late  concepts,  visualize  a  future  state,  assign 
missions,  allocate  resources  for  those  mis¬ 
sions,  assess  risk  and  make  decisions.  Dur¬ 
ing  the  fight,  commanders  see  and  under¬ 
stand  the  battlefield,  go  to  the  right  place  at 
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the  right  time,  assess  the  outcome,  and 
anticipate  change.  Commanders  lead,  guide 
and  motivate.  Command  is  commander’s 
business. 

Control  on  the  other  hand,  regulates  the 
functions  of  the  organization  and  is  a  sci¬ 
ence.  Control  is  a  more  precise  means 
through  which  staffs  support  their 
commander’s  intent  and  work  with  other 
staffs.  Control  defines  limits,  computes, 
monitors,  measures,  reports,  identifies  vari¬ 
ance  and  analyzes.  This  is  staff’s  business. 
Commanders  anticipate  change,  and  staffs 
project  change. 

While  command  focuses  the  organiza¬ 
tion,  control  regulates  the  functions  of  the 
organization.  Command  cuts  across  the 
functional  areas  at  any  given  echelon.  This 
enables  the  commander  to  use  his  intent  and 
concept  of  operations  to  focus  the  unit  and 
synchronize  the  battlefield  operating  sys¬ 
tems.  Control,  on  the  other  hand,  operates 
within  a  functional  area  and  operates  similar 
to  a  guild.  Control  depends  upon  data  and 
information  flow  up  and  down  the  functional 
area  in  a  manner  similar  to  a  stovepipe.  This 
information  flow  is  necessary,  but  remember 
the  more  control  the  less  command.  Our 
command  and  control  process  requires  com¬ 
manders  to  balance  command  and  control 
and  execute  mission  type  orders. 

This  process  is  significantly  different 
from  the  Soviet  troop  control  model.  This 
critical  path  methodology  is  normative, 
predictive,  rational,  and  easy  to  automate. 
However,  can  we  model  an  intuitive,  cre¬ 
ative,  parallel  and  sequential  process  where 
leaders  are  able  to  act  in  the  absence  of 
orders? 

Our  decisionmaking  process  is  a  continu¬ 


ous,  intuitive,  creative,  parallel  and  sequen¬ 
tial  process  —  not  an  event.  All  military 
operations  can  be  divided  into  three  phases, 
concept  formulation,  planning,  and  execu¬ 
tion  —  CPE.  Planning  and  decisionmaking 
procedures  during  a  campaign  differ  from 
those  procedures  used  before  a  campaign. 
During  battle,  leaders  need  a  streamlined 
process  to  effectively  make  decisions  in  a 
time  constrained  environment.  CPE  is  the 
three  phased  continuous  process  that  helps 
commanders  during  battle. 

Embedded  in  the  CPE  process  is  the 
leader’s  requirement  to  continuously  build, 
adjust  and  execute  plans.  While  building 
plans  leaders  need  information  that  supports 
METT-T  I^ssion,  Enemy,  Terrain,  Troops 
available-Time]  analysis  and  concept  formu¬ 
lation.  Similarly,  during  plan  adjustment  and 
execution,  information  to  leaders  must 
adequately  represent  the  situation  —  reality. 
Additionally,  leader’s  reconnaissance  must 
provide  information  that  helps  leaders  make 
adjustments  prior  to  and  during  execution. 
During  each  of  these  phases,  leaders  need 
about  right  information  in  certain  areas  to 
assist  with  timely,  right  decisions. 

Commanders  make  different  types  of 
decisions.  Commanders  determine  what  to 
allocate,  when  and  where  to  commit,  and 
what,  when,  and  where  to  engage  forces. 
These  decisions  also  differ  based  on  the 
level  of  war.  Tactical  decisions  are  mostly 
decisions  to  commit,  execute  and  engage. 
These  are  means,  space,  and  time  decisions. 

Commanders  have  many  challenges  with 
respect  to  ACE.  In  this  example,  the  com¬ 
mander  strives  to  minimize  the  delta  be¬ 
tween  committed  and  engaged  forces.  The 
challenge  is  to  know  when  to  maximize  the 
use  of  force  to  get  overwhehnmg  combat 
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power  into  the  fight  and  change  the  slope  of 
the  line.  The  commander  needs  the  right 
information  to  help  make  these  decisions. 

Decisionmaking  requires  relevant,  time¬ 
ly,  and  accurate  information.  Not  only  do 
we  need  to  know  what  information  is  re¬ 
quired,  but  we  must  also  know  where  that 
required  information  will  come  from.  That’s 
only  one  piece  of  the  equation.  We  then 
need  to  know  how  to  translate  this  informa¬ 
tion  into  decisions,  orders  and  actions.  Com¬ 
manders  can’t  wait  for  perfect  information 
before  making  a  decision.  Even  with  timely, 
relevant,  and  accurate  information,  the  prob¬ 
ability  of  making  the  right  decision  is  still 
not  guaranteed.  Decisions  must  be  decisive 
and  based  on  about  right  information.  Also, 
watch  out  for  the  somatotropin  impact.  You 
know,  that  hormone  which  affects  your  short 
term  memory. 

Sleep  impacts  on  decisionmaking.  Stud¬ 
ies  have  shown  that  without  sleep,  task 
performance  degrades  over  time.  This  is  not 
a  surprise,  but  organizing  leaders  and  staff 
to  maximize  task  performance  and  instituting 
sleep  plans  is  key.  The  conceptual  skills 
requir^  by  commanders  and  like  thinking 
battle  staffs  to  generate  concepts  are  the  first 
skills  to  degrade.  Leaders  need  rest  too.  We 
also  must  ensure  that  the  right  people,  type 
and  quantity,  are  in  the  right  jobs. 

The  commander  has  many  thoughts  but 
the  bottom  line  for  success  depends  on  his 
ability  to  make  vital  decisions.  These  deci¬ 
sions  must  not  only  be  based  on  about  right 
information,  but  must  clearly  define  what 
information  the  commander  needs,  the 
availability  of  good  information,  and  a 
means  of  providing  the  necessary  informa¬ 
tion.  Increase  the  probability  of  success  by 
using  redundant,  independent,  multiple 


phenomenology  systems  that  are  controlled 
and  available. 

Commanders  require  information  to 
support  decisionmaking  during  tactical 
operations.  Forces  accomplish  tactical  opera¬ 
tions  through  moving,  setting,  striking,  and 
recocking.  Depressing  the  probability  of 
failure  during  each  of  these  phases,  increas¬ 
es  the  probability  of  success.  By  identifying 
those  events  which  will  cause  failure,  and 
identifying  the  information  required  to 
reduce  the  risk  of  failure,  we  focus  on 
what’s  required  to  minimize  the  chance  of 
failure. 

While  trying  to  make  timely,  accurate 
decisions  on  the  current  tactical  operation, 
the  commander  is  also  faced  with  the  re¬ 
sponsibility  of  concurrently  considering 
possible  future  operations.  His  view  is 
influenced  by  his  assessment  of  the  probable 
outcome  of  the  current  operation.  Addition¬ 
ally,  the  commander  must  apply  information 
to  his  estimate  of  how  to  best  accomplish 
future  operations.  Information  must  support 
this  requirement  to  continuously  consider 
current  and  future  operations. 

Information  collection  is  a  complex 
process.  First,  consider  all  the  assets  avail¬ 
able  to  collect.  Then  look  at  all  the  nodes 
the  information  must  successfully  negotiate. 
Additionally,  the  information  must  get  to  the 
right  person.  Given  all  this  occurs,  the 
information  must  still  be  timely,  relevant 
and  accurate. 

Let’s  look  at  collection  requirements  and 
consider  all  the  conditional  probabilities.  All 
of  the  events  listed  by  the  sensor  and  the 
observable  event  must  occur  in  order  to 
simply  say  that  the  sensor  is  capable  of 
detecting  salient  targets.  That’s  a  lot  of  if. 
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and  if  statements  and  as  you  know  the  prob¬ 
ability  goes  down  as  we  work  through  an 
event.  There  will  be  holes  in  the  coverage. 
Holes  caused  by  the  sensor  and  holes  creat¬ 
ed  by  the  event  —  holes  due  to  masking, 
field  of  view,  resolution  or  swath  coverage. 
HUMINT,  in  may  cases,  is  the  only  way  to 
fill  these  Swiss  cheese  holes. 

Assuming  collection  is  possible,  this  is 
only  one  small  piece  of  the  kill  equation. 
Again,  decisions  are  required  based  on  the 
information  provided.  Information  that  sup¬ 
ports  the  decide,  detect,  and  deliver  method¬ 
ology  will  improve  decisionmaking. 

What  was  once  a  luxury  if  you  had  the 
information  is  now  a  necessity  to  successful 
combat  operations.  Commanders  must  know 
what  occurred  over  the  next  hill  —  they 
must  know  the  outcome  of  a  particular  at¬ 
tack.  BDA  helps  commanders  allocate 
means,  commit  collection  assets  and  execute 
collection  plans.  These  decisions  require 
timely,  accurate,  and  continuous  BDA. 


Tempo,  complexity,  and  lethality  wiU  aU 
increase  on  the  future  battlefield.  We  have 
less  time  to  make  decisions,  and  because  the 
battlefield  is  more  lethal,  our  decisions  have 
a  greater  consequence.  The  need  for  infor¬ 
mation  to  support  timely  and  accurate  deci¬ 
sions  will  continue.  This  is  best  supported 
by  not  continuing  to  try  to  model  our  intu¬ 
itive  decisionmaking  process,  but  by  using 
expert  systems  with  human  interface.  Work 
the  total  system,  be  ready  to  fight  blind, 
focus  on  timely,  accurate,  relevant  mforma- 
tion,  and  increase  the  chance  of  making 
right  decisions. 
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Appendix  C 

Excerpts  from  Objective  2  Plenary  Address 

Dr.  Michael  G.  Sovereign 
Naval  Postgraduate  School 


The  purpose  of  this  presentation  is  to 
provide  a  quick  review  for  you  of  the  at¬ 
tempts  to  deal  with  the  problem  of  identify¬ 
ing  C3I  measures  of  effectiveness.  The 
methodology  that  was  developed  is  called 
the  Modular  Command  and  Control  Evalua¬ 
tion  Stmcture  (MCES).  These  efforts  are 
certainly  not  the  only  efforts  that  have  been 
made  in  the  past,  but  they  do  have  a  MORS 
background  and  will  hopefully  get  you 
started  thinking  along  this  line.  .  .  . 

I  am  also  going  to  illustrate  the  proce¬ 
dure  through  an  example  of  a  system  which 
I  happen  to  be  working  on  right  now.  It  is 
the  Vertical  Take-off  and  Landing  Un¬ 
manned  Aerial  Vehicle  (VTOL-UAV).  You 
will  see  right  away  that  it  is  not  at  the 
force-level,  but  rather  at  the  system-level, 
which  is  more  in  line  with  our  effort  from 
the  previous  MORS  workshop.  From  this 
work  I  will  try  to  suggest  lessons  learned  for 
further  methodological  exploration.  .  .  . 

MCES  is  a  seven  module  process.  The 
first  module  is  Problem  Formulation.  In  this 
module  the  analyst  characterizes  the  decision 
need,  what  decisions  are  being  supported, 
what  analysis  is  required,  what  assumptions 
about  the  problem  are  required,  and  what  is 
the  mission  of  the  system  being  investigated. 
The  scenario  in  which  the  analysis  wiU  be 
conducted  must  also  be  identified  here.  .  .  . 


The  second  module  is  the  C^  system  bound¬ 
ing.  In  this  module,  we  identify  the  relevant 
entities,  both  physical  and  structural,  and  the 
boundaries  associated  with  the  system  de¬ 
scription.  These  boundaries  include  subsys¬ 
tem  descriptions,  the  C^  system,  our  own 
forces  as  well  as  enemy  forces  that  might 
interact  with  the  system.  .  .  . 

The  third  module  is  the  C^  system  defini¬ 
tion.  This  will  include  defining  the  processes 
that  fulfill  the  mission.  In  this  module  we 
will  also  map  processes  to  the  system  con¬ 
figuration.  For  the  VTOL-UAV,  the  pro¬ 
cesses  include  sensing,  assessing,  generating 
the  approaches  to  identify,  evade,  or  trail  an 
enemy,  selecting  mission  priorities,  plan¬ 
ning;  and  directing.  .  .  . 

The  fourth  module  integrates  the  statics 
and  dynamics  of  the  system  elements  with 
the  functions.  In  this  module,  we  also  devel¬ 
op  the  relationships  between  physical  enti¬ 
ties,  stmctures,  and  processes.  This  will 
provide  a  detailed  modeling  of  the  in¬ 
put/output  coupling  between  ftinctions.  The 
results  of  this  effort  will  be  an  architecture 
in  which  the  system  functions.  This  integra¬ 
tion  of  statics  and  dynamics  can  be  de¬ 
scribed  in  terms  of  entities,  structure,  and 
dynamics  at  various  levels.  For  the  UAV 
example,  it  would  like  look  this: 
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Integration  of  Statics  and  Dynamics 


Level 

Element 

Entities 

Sensor 

Structure 

Search  Pattern 

Dynamics 

Sense 

Subsystem 

Payload 

Surveillance  & 
Mission  Plans 

Assess 

Plan 

System 

UAV  & 

Mission 

Plan  Center 

CIC  &  N  IDs 
OPGENS 

Assess 
Generate 
Select,  Plan 
Direct 

Architecture 

Task  Force 

ASUW  Command 
Doctrine  & 

OP  Order 

Assess 
Generate 
Select,  Plan 
Direct 

The  fifth  module  specifies  the  measures. 
We  will  try  to  describe  the  measures  at  each 
of  the  levels  of  the  onion,  from  the  dimen¬ 
sional  parameters,  through  measures  of 
performance  and  measures  of  effectiveness, 
to  measures  of  force  effectiveness.  This  uses 
the  definitions  from  the  ’85- ’86  MORS 
Workshop.  The  measures  that  we  developed 
for  the  VTOL-UAV  are: 

1.  Dimensional  Parameters  (DP): 

a.  Weight,  size,  power,  number  of 
chaimels 

b.  Number  of  parts,  complexity, 
operational  modes 

2.  Measures  of  performance  (MOP) 

a.  Payload,  range,  endurance, 
speed,  altitude 

b.  Probability  of  detection,  sweep 
width 

c.  Pointing  accuracy,  navigational 
accuracy 

d.  Attrition  rate,  mission  success 
rate  (no  failures) 


e.  Mission  planning  time 

3.  Measures  of  Effectiveness  (MOE) 

a.  Sweep  rate,  detection  rate,  per¬ 
cent  of  targets  identified 

b.  Improvement  in  situational 
awareness 

c.  Efficiency  of  targeting 

d.  Timeliness  of  tasking  of  weapons 

e.  Time  before  enemy  detects  task 
force 

4.  Measures  of  Force  Effectiveness 
(MOFE) 

a.  Task  force  mission  success 

b.  Mission  completion  time 

c.  Ordnance  expended  per  target 
killed,  force  exchange  rate 

We  can  look  at  these  measures  in  terms 
of  the  structure  of  statics  and  dynamics. 
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Specification  of  Measures 


Level 

Element  (DP) 

Entities 

Recognitio 

Differential 

Structure 

Field  of  View 

Dvnamics 

False  Alarm  Rate 

Subsystem  (MOP) 

Pointing 

Accuracy 

Sweep  Width 

Surveillance 

Efficiency 

System  (MOE) 

Detection 

Probability 

Situational 

Awareness 

Targeting 

Timeliness 

Architecture 

(MOFE) 

Force 

Exchange 

Rate 

Task  Force 
Mission  Success 

Mission  Completion 
Time 

The  sixth  module  is  data  generation. 
This  module  identifies  the  timelines  for  data 
generation  and  how  the  data  will  be  generat¬ 
ed  —  through  an  exercise,  simulation,  ex¬ 
periment,  or  from  subject  matter  experts.  In 
this  part  of  the  analysis  process,  other  fac¬ 
tors  related  to  data  generation  are  discussed, 
such  as  reproducibility,  precision  and  accu¬ 
racy,  collection  process  and  environmental 
controls.  The  problem  is  creating  an  experi¬ 
mental  design  that  can  deal  with  scenario 
dependencies  and  having  a  man  in  the  loop 
that  is  not  unlike  his  real  command  situa¬ 
tion.  .  .  . 

The  final  module  is  the  aggregation  of 
measures  and  interpretation.  This  part  of  the 
process  is  intended  to  assemble  the  results 
and  draw  conclusions  about  the  data.  This 
should  answer  the  basic  question  —  did  the 
system  provide  a  positive  contribution. 
Some  issues  that  need  to  be  addressed  here 
are  the  scenario  dependence  of  the  results, 
causality  of  measures  —  invalidated  chain  of 
events,  sufficiency  of  measures,  and  inde¬ 
pendence  of  trials. 

Despite  community  acceptance  of  this 


methodology,  MCES  has  not  solved  the 
problem  of  credible  C^I  evaluations.  I’ve 
come  up  with  several  reasons  for  this,  and 
you  might  be  able  to  add  to  the  list.  First, 
the  MOP,  MOE,  MOFE  structure  is  hard  to 
explain,  even  to  us  analysts,  let  alone  the 
lay  person.  Second,  our  ability  to  model 
information  flow  for  decisions  is  crude  and 
lengthy.  We  can  take  an  enormous  amount 
of  time  just  deciding  what  is  aU  the  informa¬ 
tion  that  is  needed  to  make  a  credible  deci¬ 
sion.  Third,  we  have  very  few  credible 
tools  and  models  for  measuring  combat 
MOFEs.  Fourth,  in  general,  if  MOPs  are 
doubled  in  value,  the  MOEs  increase  about 
50  percent,  and  MOFEs  increase  only  1  to 
10  percent.  That’s  just  the  nature  of  the  uni¬ 
verse,  in  my  opinion.  Military  command  and 
control  systems  are  very  robust  already. 
Adding  a  little  bit  more  to  them  doesn’t 
reaUy  win  you  wars.  This  kind  of  result  is 
not  looked  upon  with  favor  by  those  selling 
C^ 

Then  how  come  we  did  so  well  with 
Desert  Storm?  My  fifth  point  is  that  trained 
people  under  motivating  conditions  can  do 
wonders  even  with  weak  systems.  Since  we 
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don’t  model  on-the-job  training,  we  didn’t 
predict  this  and  have  a  hard  time  dealing 
with  it.  Sixth,  command  and  control  will 
always  exist.  The  question  is  how  much  O'*! 
do  we  add  before  it  begins  to  get  in  the 
way.  Finally,  I  believe  that  training  and 
doctrine  are  the  most  important  parts  of  C^. 
The  rate  of  introduction  of  O'*!  may  exceed 
the  adaptability  of  these  two.  We  need  to 
keep  it  simple. 

If  that  didn’t  work  after  six  years,  where 
do  we  go  from  here?  We  should  perhaps 
explore  new  theories  of  combat  and  the  sup¬ 
porting  processes,  i.e.,  in  combat  sci¬ 
ence,  stochastic  decision  models.  We  should 


strengthen  our  joint  efforts  such  as  the  Na¬ 
tional  Training  Center,  National  Test  Bed, 
and  Defense  Modeling  and  Simulation  Of¬ 
fice  (DMSO).  We  need  to  be  able  to  re¬ 
spond  to  pressure  for  standardization,  verifi¬ 
cation,  and  validation  procedures  from 
Congress,  GAO,  and  users.  We  need  to 
explore  new  computer  tools  such  as 
object-oriented  models,  simulation  languages 
with  libraries  of  objects,  and  menu-driven 
model  building  similar  to  the  CAD/CAM 
process.  Finally,  we  need  to  develop  new 
models  of  C^,  perhaps  in  the  form  of  Petri 
Nets,  conditional  possibility,  and  variable 
resolution  modeling. 
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Appendix  D . 

Excerpts  from  Objective  3  Plenary  Address 

Dr.  Hank  Dubin 

USA  Operational  Test  and  Evaluation  Conunand 


Looking  at  measures  of  effectiveness 
from  an  operational  test  perspective  I  will 
begin  by  identifying  some  key  words  within 
the  context  of  basic  Army  definitions.  I  will 
follow  with  some  typical  measures  for  com¬ 
mand  and  control  and  discuss  how  to  teach 

which  might  be  a  way  to  get  a  hand  on 
some  measures  and  close  with  presenting 
some  challenges  and  potential  problems. 

There  are  three  classes  of  traditional 
measures  that  we  are  looking  at  to  test  the 
Army  Tactical  Command  and  Control  Sys¬ 
tem  (ATCCS):  1)  tracking  information  or 
messages  that  goes  around  the  system 
which  is  communications,  2)  following  the 
activity  that  occurs  within  the  command  post 
through  the  Army  Command  and  Control 
Evaluation  System  (ACCES),  and  3)  observ¬ 
ing  the  outcomes  of  exercises  such  as  train¬ 
ing  exercises.  .  .  . 

I  wUl  now  show  excerpts  of  the  formal, 
FM  101-5  definitions  with  emphasis  on  the 
italicized  words.  Command  and  Control  is 
a  process  where  forces  are  directed,  coor¬ 
dinated,  and  controlled  to  accomplish  the 
mission.  In  Command  there  is  authority 
vested  in  the  commander  to  do  planning  and 
directing  of  resource.  C^  Challenges  pro¬ 
vides  for  a  complex  situation  which  requires 
a  decisive  edge  over  an  opponent  which 
must  be  achieved  and  maintained.  The  C^ 
system  must  allow  for  a  great  variety  of 
tasks  and  missions  that  the  commander  must 


perform  that  require  integration  of  many 
complex  systems. 

The  tasks  that  are  to  be  accomplished 
through  the  system  consist  of  three  compo¬ 
nents:  organization,  facilities,  and  processes. 
C^  Organizations  include  the  relationships, 
authority  and  responsibility  of  the  staff  and 

functional  grouping  of  the  staff  sections. 

Facilities  require  supporting  automation 
and  communications  systems  that  are  neces¬ 
sary  for  effective  command  and  control.  C^ 
Processes  identify  the  procedures  and  tech¬ 
niques  used  to  find  out  what  is  going  on,  to 
decide  what  action  to  take,  to  issue  instruc¬ 
tions,  and  to  supervise  execution.  Another 
key  item  is  to  Get  inside  the  decision  cycle 
of  the  opponent  so  that  faster  and  better 
than  that  of  the  enemy  is  good.  The  basis  for 
the  ACCES  methodology  is  that  of  consider- 
mg  the  headquarters  as  an  adaptive  control 
system.  This  provides  for  directing  and 
monitoring  the  situation  and  allows  the  com¬ 
mander  develop  alternative  or  new  plans 
based  upon  how  adaptable  the  plan  is  when 
faced  with  a  new  assessment  of  the  situa¬ 
tion.  Measures  of  C^  are  measured  by  the 
extent  to  which  the  commander’s  intentions 
are  carried  out  and  the  ability  to  cope  quick¬ 
ly  and  effectively  with  changes  in  the  situa¬ 
tion. 

There  are  also  measures  of  outcomes 
that  identify  successful  accomplishment  of  a 
specific  mission.  They  indicate  observable 
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conditions  that  can  be  compared  with  and 
without  new  capabilities  that  support  a  pro¬ 
cess.  The  process  itself  in  not  needed  to  be 
known,  only  the  outcome  (i.e..  the  number 
of  fire  missions  accomplished  or  number  of 
aircraft  successfully  engaged).  Correlation 
of  forces  and  means  which  is  an  old  Soviet 
concept  is  the  "school  house"  solution.  In  an 
exercise  situation  to  identify  what  is  needed 
to  get  the  resources  and  capabilities  in  a 
certain  position  and  look  at  your  ability  to 
meet  the  threat.  .  .  . 

Then  there  are  some  intermediate  types 
of  measures  that  compare  to  ground  truth. 
It’s  an  assessment  to  see  how  well  we  un¬ 
derstand  various  factors  about  the  friendly 
forces,  the  enemy  forces,  the  physical  envi¬ 
ronment,  and  capabilities.  Measures  depend 
on  context,  most  are  measures  of  effective¬ 
ness  and  are  scenario  driven.  Measures  of 
task  accomplishment  must  measure  the 
ability  to  accomplish  the  mission.  It  is  one 
of  perception,  if  the  commander  thinks  its 
better  then  they  will  perform  better  and  it 
does  not  matter  what  the  "scientists"  say.  . 

The  Army  Command  and  Control  Evalu¬ 
ation  System  (ACCES)  focused  on  the  per¬ 
formance  that  can  be  observed  in  the  com¬ 
mand  post.  It  assumed  that  better  plans  can 
be  executed  with  less  need  for  modification 
and  therefore  the  plans  remain  in  effect 
throughout  their  intended  life.  ACCES  in¬ 
cludes  effectiveness  measures  as  well  as 
measures  for  accuracy  of  the  estimate  of  the 
situation.  Some  of  the  measures  are: 

1 .  Effectiveness: 

a.  Percent  of  plans  remain  in  force 
unchanged  for  the  intended  peri¬ 
od  of  time 

b.  Median  time  used  to  complete  a 


control  cycle 

c.  Percent  of  control  cycles  due  to 
major  incongruities 

2.  Monitoring: 

a.  Percent  of  information  require¬ 
ments  for  which  there  are  data 

b.  Percent  of  units  for  which  the 
most  recent  data  are  within  the 
desired  time  window 

3.  Understanding:  Percent  of  briefings 
understood 

4.  Estimating  and  Planning: 

a.  Percent  of  estimates  developed 
by  two  or  more  participants 

b.  Median  time  to  issue  a  directive 

c.  Percent  of  correct  predictions 

5.  Reporting:  Percent  of  information 
requirements  reported 

6.  Coordinating:  Percent  of  cells  in  a 
node  holding  information  aboutcide  about 
specific  units 

7.  Network:  Percent  of  nodes  holding 
information  that  coincides  about  specific 
units 

Measures  of  communications  would  be 
tracking  messages  especially  within  battle¬ 
field  functional  areas  to  see  if  the  messages 
get  to  the  right  place  at  the  right  time  and 
see  if  the  system  has  the  time.  .  .  . 

We  teach  command  and  control  in  a 
variety  of  ways.  Through  institutionalized 
training  at  our  service  schools  we  use  four 
methods  that  include  texts,  lectures,  exercis¬ 
es,  and  tests.  With  field  exercises  such  as  a 
command  post  exercise  (CPX)  we  teach  how 
to  do  things  better  through  practical  applica- 


40 


tion  within  a  simulated  combat  situation, 
then  balance  it  through  professional  readings 
and  attending  professional  symposia.  We 
give  tests  to  measure  what  is  taught.  What 
we  measure  with  these  tests  is  the  ability  to 
reason,  recall  facts,  understand  principles, 
recognize  relevant  factors.  The  assessment 
of  these  tests  comes  from  outcomes  that  are 
judged  with  respect  to  the  school  house 
solution.  .  .  . 

There  are  practical  exercises  designed  to 
test  the  four  phases  of  the  decision  making 
process.  The  first  phase  is  to  analyze  the 
mission.  Exercises  to  identify  the  purpose  of 
the  mission,  the  essential  and  implied  tasks, 
and  the  intent  of  superiors  would  be  per¬ 
formed.  .  .  . 

The  second  phase  is  to  develop  facts  and 
assumptions.  Here  we  must  develop  terrain 
and  obstacle  overlays,  identify  avenues  of 
approach,  and  consider  weather  analysis. 
We  must  also  develop  doctrinal,  situational, 
and  decision  support  templates  as  well  as 
evaluate  the  logistic  estimate.  .  .  . 

The  third  phase  is  to  develop  courses  of 
action.  Exercises  to  establish  relative  combat 
power,  array  forces,  and  develop  a  scheme 
of  maneuver  will  be  performed.  The  courses 
of  action  will  also  identify  the  means  and 
maneuver  control  measures  and  include  an 
operation  overlay  with  a  course  of  action 
statement.  .  .  . 

The  final  phase  is  to  decide  on  a  course 
of  action.  The  exercises  for  this  phase  is  to 
war  game  each  alternative  course  of  action. 
This  would  include  preparing  an  appropriate 


plan  of  action  needed  to  counter  any  course 
of  action  by  the  enemy.  .  .  . 

During  the  evaluation  of  these  practical 
exercises  we  must  determine  what  wDl  be 
measured.  It  seems  reasonable  to  measure 
the  plans  and  orders  with  respect  to  con¬ 
forming  to  Army  standards  and  quality 
(consistency,  completeness,  clarity,  and 
executabUity)  as  well  as  project  the  battle 
outcomes.  .  .  . 

There  are  three  basic  assumptions: 
More  information  is  better,  Faster  informa¬ 
tion  is  better,  and  Greater  accuracy  of  infor¬ 
mation  is  better.  These  assumptions  place  an 
enormous  burden  to  identify  the  resources 
that  are  required  to  accomplish  a  given  level 
of  effectiveness.  .  .  . 

In  conclusion,  I  must  present  some 
major  potential  problems  that  must  be  faced. 
What  do  you  do  to  measure  intent  if  the 
intent  changes?  At  a  high  level  the  intent  is 
more  Ukely  to  remain  constant  while  at  a 
low  level  intent  changes  due  to  external 
conditions.  Standard  operating  procedures 
(SOPs)  are  personality  driven.  We  must  be 
able  to  evaluate  the  techniques,  procedures, 
and  equipment  that  supports  the  decision 
making  process  without  evaluating  the 
commander’s  judgment.  Variability  due  to 
the  human  element  is  great.  Observations 
are  made  on  the  smallest  organizational  unit 
for  which  the  system  that  is  undergoing  test 
is  organic.  Testimonials  may  be  biased  by 
experience  and  background.  .  .  . 
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Appendix  £ 

Excerpts  from  Objective  4  Plenary  Address 

Mr.  Edward  C.  Brady 
Strategic  Perspectives,  Inc. 


...  I  would  like  to  offer  a  few  obser¬ 
vations  about  some  of  the  issues  you  have 
already  discussed. 

First,  I  would  like  to  give  a  different 
viewpoint  about  the  statements  that  we  better 
define  requirements,  bring  users  and  devel¬ 
opers  together  up  front  earlier  in  systems 
development  testing,  and  problems  with  the 
pace  of  technology,  etc.  I  offer  you  two 
observations:  one  is  that  the  Japanese  con¬ 
ceived  the  idea  of  the  Lexus;  designed  the 
car;  put  together  a  manufacturing  capability; 
and  produced  it,  tested  it,  and  sold  it  in  four 
years.  Most  people  think  it  is  a  pretty  damn 
good  car.  There  is  something  wrong  with 
our  process  when  many  systems  take  fifteen 
years  in  development.  While  this  is  well 
known,  I  would  argue  that  not  only  is  con¬ 
cept  development  as  difficult  or  more  diffi¬ 
cult,  but  also  it  is  as  lengthy  as  acquisition 
development.  Thus,  our  situation  is  much 
worse  than  much  debate  would  make  it 
seem.  We  spend  fifteen  years  in  concept 
development  and  we  spend  fifteen  years  in 
acquisition  and  testing,  so  it  takes  us  thirty 
years  from  when  we  think  we  need  some¬ 
thing  to  when  we  begin  to  field  the  initial 
increments  of  new  capabilities.  In  addition, 
at  least  in  the  C^I  area,  we  almost  never 
succeed  at  fielding  a  whole  system  other 
than  a  simple  thing  like  a  message  handling 
switch  at  once.  It  is  important  to  recognize 
that,  whether  we  like  it,  we  are  locked  into 
some  form  of  an  incremental  approach  not 
only  because  of  the  way  in  which  the  tech¬ 


nology  develops,  but  also  because  it  is 
inherent  in  the  basic  idea  of  command  and 
control  and  intelligence  that  we  will  con¬ 
stantly  revise  what  we  want.  We  forget  our 
own  history,  I  think,  and  that  is  why  we  get 
locked  into  some  of  these  problems. 

For  example,  I  spent  a  lot  of  time  four 
years  ago  traveling  around  the  world,  talk¬ 
ing  to  many  CINC  staffs,  talking  to  the 
chief  communications  people  in  different 
services,  and  the  J6/JCS.  Everyone  of  them 
assured  me  there  was  absolutely  no  tactical 
military  requirement  for  imagery,  NONE  — 
based  on  military  objectives,  military  strate¬ 
gy,  and  military  desire.  Today,  the  most 
pressing  need  in  many  commander’s  minds 
is  imagery.  They  not  only  want  it,  they  want 
it  real  time  at  tactical  maneuver  level.  .  .  . 
This  type  of  sudden  change  in  requirements 
is  the  norm.  Thus,  I  don’t  see  how  there  is 
any  hope  at  all  in  the  kind  of  arena  we  work 
in  of  having  any  rigid,  long-term  standing 
requirement  for  a  system  —  even  if  we 
understood  militarily  what  commanders 
today  think  they  want  to  do.  That  is  not  to 
say  that  we  shouldn’t  put  effort  into  defining 
existing  requirements,  but  rather  that  we 
also  have  to  find  a  method  that  accommo¬ 
dates  fluid  requirements,  a  process  that  lets 
us  implement  them  in  stages,  and  field 
increments  quickly.  Any  approach  that 
doesn’t  contribute  to  such  an  end  will  not  be 
useful. 

As  a  second  example,  when  I  was  at  The 
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MITRE  Corporation,  I  managed  through  the 
period  when  we  first  introduced  personal 
workstations.  We  went  through  four  genera¬ 
tions  of  workstations  in  nine  years.  .  .  .We 
started  a  program  that  said  while  we  are 
changing  what  we  are  buying,  we  also  will 
reprioritize  who  has  what..  .  .  The  Com¬ 
manders  in  the  field  will  never  be  satisfied 
if  we  don’t  find  a  process  that  works  like  I 
have  described.  They  will  always  want 
something  different  than  a  few  years  before, 
they  will  always  say  what  we  are  fielding  is 
not  what  they  want.  They  will  always  say 
we  are  too  slow.  They  will  always  have  test 
beds  off  on  the  side,  finding  something  that 
uses  newer  technology  than  the  formal 
program,  and  replacing  whatever  system  it 
is  —  MCS,  AS  AS  —  whatever  system  we 
have  been  developing  through  a  logical 
process.  It  is  just  in  the  nature  of  it.  It  is  in 
the  nature  of  the  underlying  technologies.  . 
.  So,  we  must  find  processes  which  accom¬ 
modate  these  inherent  dynamics  and  not  sit 
around  and  wish  they  were  different. 

...  I  would  draw  your  attention  to  LTG 
Shoffner’s  constant  references  to  the  acceler¬ 
ation  in  the  pace  of  battle.  What  this  means 
in  terms  of  C^I  systems  is  constantly  revised 
information  processing  requirements.  Every 
tune  that  we  can  process  information  faster, 
the  commanders  will  want  to  change  how 
they  fight  and  ask  that  we  find  ways  to 
process  information  even  faster.  That  is  the 
nature  of  our  business.  ...  As  soon  as  you 
start  to  exploit  the  benefits  of  automation, 
you  change  the  functionality  of  the  process 
it  supports.  As  soon  as  you  change  the  func¬ 
tion,  you  see  new  ways  to  change  the  re¬ 
quirement,  and  you  change  the  way  you 
want  to  fight.  ...  As  long  as  these  dynam¬ 
ics  are  tme  in  command  and  control,  intelli¬ 
gence,  and  electronic  warfare,  we  have  to 
deal  with  requirement  changes  during  devel¬ 


opment  of  systems.  We  have  to  deal  with 
them  in  parallel  with  technology  changes. 
We  have  to  have  a  process  that  supports 
both  types  of  change. 

I  would  now  like  to  talk  a  bit  about 
MOEs.  A  lot  of  you  know  that  I  was  a  par¬ 
ticipant,  along  with  a  lot  of  people  in  this 
room,  in  the  work  that  went  on  in  the  early 
’80s  which, -in  my  view,  largely  culminated 
in  the  MORS  workshop  report  that  you  were 
given  in  the  read  ahead  package.  .  .  .  This 
effort  baselined  most  of  the  extant  methodol¬ 
ogy,  a  lot  of  models,  surveyed  what  the 
community  had  and  didn’t  have.  It  made 
recommendations  of  what  we  should  do  next 
and  began  to  fund  research  efforts  to  push 
us  forward.  .  .  . 

.  .  .The  most  important  features  of  that 
period  were  that  there  was  a  known  commu¬ 
nity  of  interest,  there  was  an  interest  and 
desire  in  funding  activities,  and  some  prog¬ 
ress  was  being  made.  You  can  argue  about 
whether  it  was  fast  or  whether  it  was  slow, 
that  is  not  my  point.  My  point  is  that  those 
involved  had  an  increasing  commonalty  of 
definitions  and  an  increasing  consensus  of 
what  we  had  and  what  we  didn’t  have  and 
an  increasing  sense  of  where  we  were  going. 

One  of  the  things  I  read  on  the  plane  last 
night  coming  here  was  the  paper  by  Ingram 
and  Short.  While  that  paper  has  much  to 
commend  it,  I  was  struck  by  what  I  think 
was  the  basic  factor  of  that  paper.  It  went 
back  to  several  original  works  and  said  we 
needed  to  take  a  step  forward.  The  question 
I  put  to  myself  was  why  are  we  only  taking 
a  step  forward  seven  years  later?  What  have 
we  been  doing  in  between?  What  has  hap¬ 
pened  to  the  continuity  of  effort,  the  willing¬ 
ness  to  fund  research,  and  everything  else? 
Based  on  less  than  comprehensive  observa- 
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tions,  I  would  answer  that  we  have  done 
some  good  things  in  between,  but  we  have¬ 
n’t  done  enough.  We  have  lost  too  much  of 
the  continuity  of  effort.  This  forum  is  im¬ 
portant  in  the  sense  that  it  brings  people 
back  to  the  community  and  refocuses  us  on 
a  problem  area. 

It  is  particularly  apropos  to  do  this  in 
light  of  the  ferment  going  on  amongst  the 
military  about  how  do  they  want  to  fight, 
what  kind  of  systems  do  they  need,  even 
what  is  their  mission.  ...  we  are  seeing  a 
resurgence  in  intellectual  thinking  about 
combat  and  our  national  role  in  it.  The 
pressing  need  to  redefine  the  role  of  the 
military  m  the  post  cold  war  era  has  pro¬ 
duced  a  great  amount  of  very  challenging, 
exciting,  interesting  thinking.  ...  All  this 
provides  our  analytical  community  an  oppor¬ 
tunity  to  plug  into  and  take  advantage  of  this 
ferment,  regain  our  former  focus  and  em¬ 
phasis  on  measures  of  effectiveness;  our 
interest  in  the  theory  of  command  and  con¬ 
trol  and  how  we  can  experiment,  measure, 
and  modify  architectures  and  systems.  .  .  . 
This  workshop  offers  us  a  good  opportunity 
to  look  at  these  issues. 

.  .  .  Since  I  recently  have  been  spending 
a  lot  of  time  with  the  Army  and  we  are  at 
an  Army  installation,  I  will  use  some  Army 
examples  to  illustrate  some  thoughts  I  want 
to  share  with  you.  I  don’t  mean  to  pick  on 
the  Army  in  any  way.  ...  At  Fort  Sill,  the 
Army  Science  Board  meeting  I  was  attend¬ 
ing  had  a  briefing  on  a  system  called 
ATACMS,  which  is  a  long-range  Army 
missile.  It  was  a  very,  very  nice  briefing  — 
excellent  graphics,  easy  to  understand,  lots 
of  computer  icons,  lots  of  color  —  all  used 
to  portray  an  interesting  message.  It  used  all 
the  latest  tools  in  multimedia  presentations. 
It  also  had  good  analytical  substance  to  it. 


The  major  thrust  was  —  so  what  is  the 
case  for  buying  ATACMS?  Why  should  we 
buy  it?  Of  course,  the  answer  was  already 
known  since  it  is  an  Army  system  we  long 
have  wanted  to  buy.  It  was  good  advocacy 
analysis.  In  the  course  of  the  analysis,  they 
did  what  you  all  know  is  the  right  thing  to 
do  —  they  made  extensive  use  of  sensor 
target  coverage  footprints  and  timeline  com¬ 
parisons,  which  then  required  them  as  they 
looked  at  system  capabilities  to  look  at  com¬ 
mand  and  control  systems.  .  .  .  They  used 
very  good  measures  of  performance,  mea¬ 
sures  of  effectiveness  and,  they  did  good 
quantification  of  them  with  very  simple 
arithmetic. 

What  I  would  like  to  convey  to  you 
about  this  study,  however,  is  that  we  did  the 
same  study  fifteen  years  ago,  the  exact  same 
concq)tual  system,  exact  same  sensors,  and 
surprisingly  enough  reached  the  same  con¬ 
clusions  using  the  same  measures.  The 
people  who  worked  on  this  study  did  not 
know  this.  My  suspicion  is  .  .  .  that  they 
had  about  ninety  to  one  hundred  twenty  days 
to  quickly  respond  to  a  higher  headquarters’ 
need  to  do  some  analysis  that  shows  the 
utility  of  ATACMS.  They  are  bright,  good 
people.  They  did  a  lot  of  good  things  in  the 
study,  but  they  didn’t  build  on  anything  that 
had  already  been  done,  and  they  didn’t 
know  they  were  reconfirming  things  that 
already  had  been  analyzed.  .  .  . 

Another  thing  that  struck  me  about  the 
read-ahead  paper  ...  is  the  call  for  stan¬ 
dardization.  I  happen  to  personally  be 
schizophrenic  about  standardization.  I  want¬ 
ed  to  share  this  bias  with  you  to  help  you 
understand  what  I  am  about  to  say.  Stan¬ 
dards  help  us  in  many,  many  ways.  How¬ 
ever,  they  hinder  us  in  two  important  ways. 
First  of  all,  once  we  have  standardized,  if 
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we  haven’t  included  everything  we  need  to 
include  in  our  changing  environment,  it  is 
very  difficult  to  reach  agreement  to  change 
the  standards  —  and  so  the  standards  be¬ 
come  inhibitors  once  we  agree  to  them. 
Second,  most  people  who  issue  a  call  for 
standardization,  and  I  may  have  misunder¬ 
stood  the  paper  in  this  sense,  seem  to  be 
saying  that  if  we  could  just  comprehensively 
a  priori  define  all  the  measures  we  need  and 
standardize  them  we  would  not  have  any 
more  problems.  ...  To  think  that  we  can 
do  this  is  a  normal,  top-down  analytical  way 
of  looking  at  things.  We  need  to  be  working 
on  attempts  to  do  so,  but  to  hold  up  prag¬ 
matic  attempts  to  move  forward  in  a 
bottom-up  way  while  we  are  waiting  for 
such  efforts  to  succeed  would  be  a  crucial 
mistake.  We  need  to  move  in  both  directions 
in  parallel  and  accept  that  we  will  be  doing 
so  forever.  There  are  a  lot  of  good  things 
we  can  do  now  with  what  we  do  have  as 
measures,  what  we  do  have  as  data,  and  for 
what  we  do  need  to  do  daily  to  support  the 
various  decisions  for  which  analysis  is  used. 
What  we  usually  fail  to  do  is  to  buUd  these 
partial  efforts  continuously  into  a  framework 
that  lets  us  get  closer  to  the  whole.  Instead, 
we  do  almost  everything  incrementally. 
And,  do  it  discontinuously.  We  do  it  without 
knowledge  of  what  each  other  is  doing.  We 
do  not  keep  integrating  and  building,  and 
stepping  back  from  our  efforts  and  taking 
top-down  assessments  of  where  we  are. 
Where  have  we  already  filled  in  the  blanks? 
What  blanks  still  exist?  What  new  ones  have 
come  along? 

This  brings  the  discussion  back  to  two  of 
the  themes  I  wanted  to  convey  to  you  —  the 
lack  of  a  forum,  and  the  lack  of  a  research 
thrust  to  what  we  are  doing.  While  thinking 
about  this,  I  was  struck  by  a  talk  show  I  saw 
on  TV  this  morning  about  the  presidential 


debates.  .  .  .  The  panel  said  that  the  prob¬ 
lem  with  the  debates  is  that  the  questions 
weren’t  draw  up  really  well.  They  weren’t 
focused  enough.  .  .  .  They  gave  too  much 
leeway  to  the  candidates.  .  .  An  individual 
in  the  audience  stood  up  and  said  they  didn’t 
think  that  was  the  problem  at  all.  They 
thought  that  the  problem  was  the  answers 
the  candidates  gave!  I  think  we  ought  to 
examine  ourselves  from  that  point  of  view. 
Are  we  really,  in  whatever  increment  of 
effort  we  are  doing,  defining  rigorously 
what  measures  we  want  to  use  and  why;  and 
are  they,  in  fact,  applicable  to  the  analysis 
we  are  conducting  or  the  test  we  are  doing 
or  whatever  decision  we  are  supporting?  Are 
they  judiciously  chosen?  Do  we  know  how 
to  measure  them,  and  if  we  know  how  to 
measure  them  what  data  do  we  need  to 
collect  to  do  that?  And,  if  we  are  going  to 
collect  that  kind  of  data,  what  instrumenta¬ 
tion  do  we  need  to  collect  it?  I  think  we 
need  to  look  hard  at  the  kind  of  answers  we 
would  be  giving  to  these  questions.  There  is 
a  lot  of  poor  quality  work  done  by  us.  I 
don’t  know  the  reason  for  that,  other  than 
maybe  time  and  resources,  and  a  failure  to 
use  earlier  work  properly.  We  have  excel¬ 
lent  studies  and  we  have  very  poor  studies, 
and  they  seem  to  be  done  side  by  side.  You 
can  go  around  the  community  at  any  given 
time  and  find  this.  ...  It  isn’t  unique  to 
particular  organizations  or  particular  efforts. 
I  think  we  are  not  hard  enough  on  ourselves 
about  what  are  the  answers  as  opposed  to 
what  we  are  missing  in  the  form  of  ques¬ 
tions.  We  don’t  hold  ourselves  responsible 
enough  for  making  continuous  progress  in 
defining  measures,  using  them  judiciously, 
and  using  them  consistently  —  even  if  we 
don’t  know  what  they  all  are,  even  if  we 
don’t  have  standards  for  all  of  them,  and 
even  if  we  don’t  have  aU  the  data  we  need. 
Thus,  I  think,  from  a  ground-up  way  of 
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operating  —  we  could  make  a  lot  of  prog¬ 
ress  towards  what  we  could  define  from  a 
top-down  need. 

I  want  to  talk  a  little  about  technologies 
and  tools.  .  . 

I  think  that  the  advances  we  are  making 
in  instrumentation  are  quite  phenomenal  and 
are  very  exciting.  We  could  soon  collect 
rooms  and  rooms  and  rooms  of  data  and  we 
may  do  that.  What  I  am  worried  about  is 
whether  we  are  going  to  do  that  to  no  end, 
or  whether  we  know  what  we  are  doing.  For 
example,  the  White  Sands  Missile  Range  has 
done  a  great  job  of  improving  instrumenta¬ 
tion  over  the  last  five  or  six  years.  The  Fort 
Huachuca  Electronic  Proving  Ground  is  now 
proceeding  pell-meU  to  implement  extensive 
automated  instrumentation  for  SINCGARS 
and  EPLRS  testing  —  because  they  were  not 
allowed  to  pursue  their  instrumentation  plan 
of  ten  years  ago.  So  it  is  often  in  the  case  of 
our  business.  Not  because  we  do  not  have 
smart  people.  Not  because  they  do  not  think 
ahead  about  what  they  wiU  need.  But,  rather 
because  of  the  way  that  budgets,  approval 
processes,  and  everything  else  works  —  you 
sort  of  get  things  just  in  time  rather  than 
ahead  of  time. 

When  you  get  instrumentation  just  in 
time,  then  you  learn  how  to  use  it  and  de¬ 
fine  what  you  want  to  do  at  the  last  minute 
—  thus,  we  make  a  lot  of  mistakes.  Again, 
I  think  that  this  is  inherent  in  the  nature  of 
things.  .  .  .  But  it  is  important  to  remember 
that  we  can  also  say  —  How  can  we  build 
on  what  we  did  do?  How  can  we  learn  from 
what  we  have  now?  How  can  we  do  the  next 
test  better?  How  can  we  take  advantage  of 
what  the  Navy  already  did,  or  how  can  we 
take  advantage  of  what  the  Army  already 
did?  We  can  only  improve  ourselves  in  this 


way  through  some  kind  of  community  forum 
that  meets  periodically  and  discusses  these 
things. 

There  wiQ  be  a  lot  of  testing  going  on 
over  the  next  five  or  six  years  even  though 
we  are  now  buying  a  lot  less.  Many  major 
C^I  systems  are  just  entering  early  testing, 
many  will  be  going  through  product  im¬ 
provement,  many  will  be  candidates  for 
significant  technology  insertion  programs.  . 
.  .  How  are  we  going  to  know  that  changes 
being  proposed  for  systems  will  reaUy  im¬ 
prove  it  overall  as  a  system?  How  do  we 
know,  if  we  don’t  set  up  an  analytical  con¬ 
struct,  if  we  change  the  communications 
links  to  do  direct  fire  with  imagery  to 
ATCCMS  what  the  impact  is  on  the  rest  of 
the  system.  We  have  no  practical  way  to  get 
it  all  out  in  the  field  at  once  and  test  it. 

This  brings  me  to  topics  like  simulation. 
Many  of  you  know  that  I  have  spent  a  lot  of 
time  over  the  last  four  years  on  interactive 
simulation  concepts.  I  assume  most  of  you 
know  about  a  DARPA  program  that  they  are 
trying  to  get  off  the  ground  called  Synthetic 
Battlefield  Environment  on  Demand 
(SYNBED).  This  is  an  effort  to  network 
ranges  together,  generally  in  the  Southwest 
United  States  —  Nellis,  parts  of  White 
Sands,  maybe  Twenty-Nine  Palms,  a  Navy 
thing  yet  undefined,  and  the  Army  National 
Training  Center.  Doing  this  in  a  way  m 
which  the  network  would  be  compatible  with 
the  efforts  in  the  Distributed  Interactive 
Simulation  area,  you  would  allow  a  ground 
commander  to  have  an  actual  battalion  in 
attack  at  the  NTC  and  an  air  commander  to 
have  actual  aircraft  coming  in  and  support¬ 
ing  that  battalion  with  close  air  support  out 
of  Nellis;  but  also  have  an  overall  air  com¬ 
mander  fighting  not  only  the  close  air  sup¬ 
port  battle  but  also  fighting  a  whole  air 
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campaign  which  is  filled  in  with  simulation. 
Similarly,  a  ground  commander  would  be 
not  only  fighting  the  battalion  in  attack,  but 
also  fighting  a  simulated  whole  division. 
Also,  the  simulation  forces  could  be  paced 
by  the  battalion  that  is  actually  on  the 
ground  and  the  aircraft  supporting  it.  In 
addition,  one  could  include  all  the  forces  on 
the  flank,  rear  areas,  logistical  supplies,  etc. 

I  wanted  to  bring  this  up  because,  as¬ 
suming  that  we  are  going  to  succeed  at 
implementing  this  effort,  we  will  have  a 
great  environment  rich  with  C^,  intelligence, 
and  electronic  warfare  activities  and  many 
echelons  of  command  —  which  we  do  not 
normally  have  at  our  disposal.  We  wiU  have 
an  enormous  range  of  opportunity  for  imple¬ 
menting  instrumentation.  We  can  practically 
put  in  data  collection  probes  anywhere  we 
like.  Much  of  the  activity  is  simulated,  yet, 
we  also  will  have  a  lot  of  humans  in  the 
loop.  The  question  to  ask  ourselves  is  are 
we  ready  to  do  that?  What  are  we  going  to 
do  with  all  that  data?  And,  how  will  we 
know  what  it  means? 

...  I  think  the  analysts  need  to  get  a 
conceptual  framework  for  their  work  in 
place  quickly.  We  are  going  to  be  using 
many  new  technologies  for  a  lot  of  things. 
We  are  going  to  be  using  these  capabilities 
for  training,  using  them  for  concept  devel¬ 
opment,  using  them  for  system  definition, 
electronic  prototyping,  to  carry  out  part  of 
the  testing.  What  are  we  in  our  analytic 
community  going  to  do  about  using  all  this 
as  an  opportunity  to  collect  data,  to  measure 
things,  to  test  hypotheses  we  have  about 
what  is  better  than  what,  about  how  we 
know  when  we  have  improved  command 
and  control  and  inteUigence? 

We  now  have  on  the  battlefield  an  enor¬ 


mous  number  of  high  speed,  high  quantity 
collectors  of  data  —  sensors.  They  are 
dumping  data  all  over  the  place.  What  we 
are  struggling  with  at  this  point  is  to  define 
what  pieces  of  data  and  information  are 
relevant  to  whom,  and  where  do  we  want 
data  dumped  when.  We  have  a  lot  of  tech¬ 
nology  moving  into  the  field  very  quickly 
that  gives  us  great  flexibility  in  doing  this. 
We  have  a  lot  of  technology  becoming 
available  that  the  services  will  presumably 
buy  for  training  purposes  and  testing  purpos¬ 
es  that  wUl  provide  us  with  a  lot  of  data. 

We  could  gain  a  lot  of  insight  into  many 
issues  if  we  had  a  broad  framework  of  what 
it  is  we  want  to  know,  and  what  it  is  we 
want  to  measure,  and  what  our  hypotheses 
are  about  those  things,  and  how  we  can  go 
about  collecting  the  data  to  prove  or  dis¬ 
prove  those  hypotheses,  and  to  quantify 
those  measures.  I  understand  that  you  have 
had  a  briefing  on  the  art  of  conunand  and 
the  science  of  control.  While  I  agree  with 
the  general  thrust  of  this,  I  certainly  don’t 
conclude  that  we  shouldn’t  measure  the  art 
part  of  command  and  control.  Our  measure¬ 
ments  here  may  be  more  subjective,  they 
may  be  more  difficult;  but  it  certainly  isn’t 
something  we  should  stop  doing! 

.  .  .1  would  like  to  summarize  by  empha¬ 
sizing  that  my  own  experience  in  this  com¬ 
munity,  as  a  participant  in  combat  opera¬ 
tions,  as  a  national  level  intelligence  analyst, 
and  an  R&D  worker  in  acquisition,  all  leads 
me  to  the  same  conclusion.  We  have  a  pro¬ 
cess  of  funding  and  tasking  that  is  inherently 
not  desirable  from  a  logical,  analytical  point 
of  view.  We  get  things  at  the  last  minute. 
We  are  tasked  to  do  things  quickly.  We 
don’t  get  the  resources  when  we  ask  for 
them,  but  rather  we  get  them  at  the  last 
minute.  We  also  have  an  environment  of 
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warfighting  and  technology  —  both  of  which 
rapidly  evolve.  Yet  I  believe  that  we  can 
make  lots  of  progress  if  we  put  our  minds  to 
it.  We  can  have  a  forum.  We  do  have  some 
amount  of  money  going  into  research  ef¬ 
forts.  We  can  talk  to  one  another  and  share 
important  information.  We  can  build  on  the 
many  things  that  are  happening.  By  doing 
more  of  all  of  this,  I  believe  we  can  make  a 
lot  of  progress. 
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Appendix  J 
Acronyms 


ACCES 

AES 

ALSP 

ARI 

ATACMS 

ATCCS 

BCTP 

BDA 

BEIT 

C^FAM 

C^IEW 

CAD 

COEA 

DMSO 

DoD 

EAC 

EEA 

EW 

HEAT 

MCES 

METT-T 

MNS 

MOE 

MOPE 

MOM 

MOP 

MORS 

MTM 

NTC 

OPVIEW 

ORD 

PACOM 

PIR 

SAMS 

SME 

SOP 

SYNBED 

TACFIRE 


Army  Command  and  Control  Evaluation  System 

ATCCS  Experimentation  Site 

Aggregate  Level  Simulation  Protocol 

Army  Research  Institute 

Army  Tactical  Missile  System 

Army  Tactical  Command  and  Control  System 

Battle  Command  Training  Program 

Battlefield  Damage  Assessment 

Battle  Force  In-Port  Training 

Command  and  Control  Functional  Area  Model 

Command,  Control,  Communications,  Intelligence,  and  Electronic  Warfare 

Computer  Aided  Design 

Cost  and  Operational  Effectiveness  Analysis 

Defense  Modeling  and  Simulation  Office 

Department  of  Defense 

Echelons  Above  Corps 

Essential  Elements  of  Analysis 

Electronic  Warfare 

Headquarters  Effectiveness  Assessment  Tool 

Modular  Command  and  Control  Evaluation  System 

Mission,  Enemy,  Terrain,  Troops,  and  Time 

Mission  Need  Statement 

Measure  of  Effectiveness 

Measure  of  Force  Effectiveness 

Measure  of  Merit 

Measure  of  Performance 

Military  Operations  Research  Society 

Model-Test-Model 

National  Training  Center 

Operational  Value  of  Intelligence  and  Electronic  Warfare 

Operational  Requirements  Document 

US  Pacific  Command 

Priortized  Intelligence  Requirements 

School  of  Advanced  Military  Studies 

Subject  Matter  Expert 

Standard  Operating  Procedure 

Synthetic  Battlefield  Environment  on  Demand 

Tactical  Fire  Direction  System 


59 


TCDC 

TEMP 

TOS 

TRAC 

TRADOC 

USACGSC 

UAV 

VIC 

VTOL 


Tactical  Commander  Development  Course 

Test  and  Evaluation  Master  Plan 

Tactical  Operations  System 

TRADOC  Analysis  Command 

Training  and  Doctrine  Command  (US  Army) 

US  Army  Command  and  General  Staff  College 

Unmanned  Aerial  Vehicle 

Vector  in  Commander 

Vertical  Takeoff  and  Landing 
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